Large language models (LLMs) have transformed AI research thanks to their powerful internal capabilities and knowledge. However, existing LLMs still fail to effectively incorporate the massive external knowledge when interacting with the world. Although retrieval-augmented LLMs are proposed to mitigate the issue, they are still fundamentally constrained by the context length of LLMs, as they can only retrieve top-K raw data chunks from the external knowledge base which often consists of millions of data chunks. Here we propose Thought-Retriever, a novel model-agnostic algorithm that helps LLMs generate output conditioned on arbitrarily long external data, without being constrained by the context length or number of retrieved data chunks. Our key insight is to let an LLM fully leverage its intermediate responses generated when solving past user queries (thoughts), filtering meaningless and redundant thoughts, organizing them in thought memory, and retrieving the relevant thoughts when addressing new queries. This effectively equips LLM-based agents with a self-evolving long-term memory that grows more capable through continuous interaction. Besides algorithmic innovation, we further meticulously prepare a novel benchmark, AcademicEval, which requires an LLM to faithfully leverage ultra-long context to answer queries based on real-world academic papers. Extensive experiments on AcademicEval and two other public datasets validate that Thought-Retriever remarkably outperforms state-of-the-art baselines, achieving an average increase of at least 7.6% in F1 score and 16% in win rate across various tasks. More importantly, we further demonstrate two exciting findings: (1) Thought-Retriever can indeed help LLM self-evolve after solving more user queries; (2) Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.
Figures & tables
Figure 1: Why Thought-Retriever helps . (a) A standard RALM is limited by the number of retrieved chunks. The retrieved data fails to cover all the necessary data chunks (red chunks) for a user query. (b) A hierarchical RALM retrieves summaries Si , generated independently from user queries, which could improve recall at the cost of lower precision. (c) Thought-Retriever leverages past LLM thoughts collected from answering user queries, with little computational overhead. Thought-Retriever balances low-level facts and high-level thoughts, leading to high precision and recall.
Figure 2: Thought-Retriever Framework . (a) Thought retrieval: Upon receiving a user query, Thought-Retriever retrieves top-K data chunks from the mixture of external knowledge and thought memory based on embedding similarity; (b) Answer and confidence generation: The LLM generates the answer for the user query based on the retrieved data chunks; (c) Thought generation: The LLM further generates thoughts and its confidence based on the user query and the generated answer; (d) Thought merge: The calculation of similarity is used to measure whether the generated thought will cause redundancy in data chunks; (e) Thought memory update: Meaningless and redundant thoughts are removed and the remaining novel thoughts are used to update the thought memory.
Figure 3: Thought and Confidence Generation Prompt. This prompt is used for Thought and Confidence Generation as described in Section 2.3 . It evaluates whether the answer is valid and meaningful, and then summarizes the query and answer into a thought.
Dataset
Task Type
Avg. len
Cases
AcademicEval
Abstract-single
Single Sum
8,295
100
Abstract-multi
Multi Sum
33,637
30
Related-multi
Multi Related
22,107
30
Public Datasets
Gov Report
Single QA
8,910
100
Table 1: Overview of Datasets: task types, average length, and number of cases.
Type
AcademicEval
Public
Dataset
Abstract-single
Abstract-multi
Related-multi
Gov Report
WCEP
Method
F1
Win Rate
F1
Win Rate
F1
Win Rate
F1
Win Rate
F1
Win Rate
BM25
0.212
7%
0.232
7%
0.203
40%
0.211
30%
0.178
31%
TF-IDF
0.202
4%
0.225
4%
0.207
40%
0.195
35%
0.223
34%
Contriever
0.242
13%
0.232
15%
0.201
35%
0.223
40%
0.211
40%
DPR
0.206
4%
0.226
4%
0.196
30%
0.188
20%
0.201
33%
Table 2: Thought-Retriever consistently outperforms all the baselines in fact retrieval datasets . Bold and underline denote the best and second-best results. F1 score evaluates the similarity with the ground truth, higher is better. Win Rate represents the frequency with which a method’s response is preferred over Thought-Retriever by the evaluator. The 50% entry for Thought-Retriever serves as the reference point (a tie); consequently, for other baselines, a win rate lower than 50% indicates that they are outperformed by Thought-Retriever. Note that the maximum context length is 2,000 tokens for all retriever-based methods and Thought-Retriever employs Contriever as its retriever.
Setting
Retriever-origin
Response-direct
Retriever-other
Thought-Retriever
F1
0.25
0.19
0.22
0.24
Table 3: Thought-Retriever can help the LLM quickly learn from other LLMs . Retriever-origin is the golden setting that retrieves original facts, others are comparative settings without original facts.
Figure 4: Thoughts from other LLMs help respond without fact. It presents an illustrative example in which our LLM communicates with four other LLMs, each an expert in a different field. These expert LLMs are assigned specific roles (e.g., doctor) with different background knowledge. Our LLM is then able to rapidly learn from their thoughts and incorporate them as external knowledge.
Figure 5: Deeper thoughts help abstract queries. This figure illustrates the correlation between six questions, categorized by their level of abstraction as evaluated by expert LLM (x-axis), and the abstraction level of the corresponding retrieved information (y-axis). The questions are grouped into three categories: high abstraction (top 2 questions), medium abstraction, and low abstraction. Keywords from each question are displayed next to their corresponding data points for clarity.
Figure 6: Thought-Retriever can indeed help LLM self-evolve after solving more user queries. It illustrates that the performance of LLM across two datasets shows an upward trend as the number of thoughts increases.
Figure 7: Contriever and thoughts filtering are suitable for Thought-Retriever. Ablation study of 6 methods on two datasets helps us decide on important components of Thought-Retriever.
Figure 8: Thought-Retriever performs better in balancing recall and precision. The dotted line indicates the exact balance between precision and recall. The closer the dotted line is, the better the balance is.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Attribute
Description
Abstract-Single
‘title’
The title of the academic paper.
‘abstract’ as label
The abstract of the academic paper.
‘main_content’
The content of the paper excluding the abstract and the conclusion.
Abstract-Multi
‘title 1’
The title of the first academic paper.
‘abstract 1’
The abstract of the first academic paper.
‘main_content 1’
The content of the first paper excluding the abstract and the conclusion.
Appendix
Table 4: AcademicEval Dataset Documentation. This table presents the specific format of the data in our AcademicEval dataset.
Figure 9: AcademicEval Usage Instructions. This figure provides a visualization of the usage instructions for the AcademicEval dataset, as described in Section A.2 , to aid understanding.
Figure 10: Prompt for Writing Abstracts. This prompt was used in our experiment to ask the LLM to write an abstract based on the retrieved information. We provided in-context instructions to guide the LLM in producing higher-quality responses.
Figure 11: Abstract Multi Ground Truth Prompt. This prompt was used in our experiment on the Academic-abstract-multi dataset. Specifically, for each data entry, we summarize the abstracts of five papers in the entry to create the ground truth. To ensure high-quality generation, we utilized GPT-4o as the expert LLM to synthesize these summaries based on the provided prompt.
Figure 12: Prompt for Writing Related Works. This prompt was used in our experiment to ask the LLM to write a related work section based on the original paper’s abstract and the retrieved related materials. We also provided an example of in-context learning to enable the LLM to perform more effectively on this challenging task.
Figure 13: User Query Formation Prompt. This figure presents the prompt used to model real-world user queries. Specifically, it includes two methods: template-based query formation, where general question templates are created to be suitable for a wide range of papers, and LLM-based query formation, where this prompt is used to ask an LLM to generate diverse queries.
Figure 14: AI Evaluation Prompt. This prompt is used for the AI Evaluation metric Win Rate, as described in Section 4 . Given two generated answers and the ground truth answer, we ask the expert LLM to determine which generated answer aligns more closely with the ground truth.
Query
What has driven significant progress in various NLP tasks in recent years?
Response
According to the text, significant progress in various NLP tasks in recent years has been driven in part by benchmarks such as GLUE, whose leaderboards rank models by how well they perform on these diverse tasks.
Prompt template
Input: Given query:{query}, given response:{response}. Based on the provided query and its corresponding response, perform the following step: succinctly summarize both the question and answer into a coherent knowledge point, forming a fluent passage.
Thought candidate
Here is a summarized knowledge point: In recent years, significant progress has been made in various Natural Language Processing (NLP) tasks. A key driver of this progress is the development of benchmarks, such as GLUE, which provide a standardized way to evaluate and compare the performance of different models on a range of diverse NLP tasks. These benchmarks, which often take the form of leaderboards, rank models based on their performance, fostering competition and innovation in the field. As a result, researchers and developers have been motivated to improve their models, leading to significant advancements in NLP capabilities.
Appendix
Table 5: Example of thought construction . For a given query, Thought-Retriever retrieves the corresponding data chunks (thoughts and original data chunks) from the knowledge corpus to get the response. Then, Thought-Retriever integrates the query and response into a prompt through the prompt template and obtains the final thought candidate through the output of LLM. This thought candidate is then evaluated to determine whether it is correct and not redundant.
Abstraction
Rank (expert LLM)
Query
High
6 (Most Abstract)
"What are the broader future implications of user-centric utility in NLP model evaluation?"
High
5
“Please craft an abstract summarizing the key points from the provided text.”
Medium
4
"What are some of the limitations of this study?"
Medium
3
“What are the key methods introduced in this paper?”
Low
2
"Please explain the term Minerva to me."
Low
1 (Least Abstract)
"How many benchmarks are used to test the model’s long context understanding ability in this paper?"
Appendix
Table 6: Sample Queries Used in Abstraction Level Case Study. This table presents sample queries from the case study conducted in Section 4 , which demonstrates how Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.
Type
Abstract-single
Abstract-multi
LLM
Qwen-7b
Llama-3-70b
Qwen-7b
Llama-3-70b
Method
F1
Win Rate
F1
Win Rate
F1
Win Rate
F1
Win Rate
BM25
0.196
3%
0.22
13%
0.232
7%
0.233
14%
TF-IDF
0.192
3%
0.21
17%
0.220
3%
0.224
12%
Contriever
0.231
10%
0.238
18%
0.229
4%
0.228
19%
DPR
0.209
4%
0.215
18%
0.222
3%
0.222
11%
Appendix
Table 7: Thought-Retriever demonstrates adaptability across different LLMs . This table compares Thought-Retriever’s performance against baselines on Abstract-single and Abstract-multi tasks using Qwen-7B and Llama-3-70B models. Thought-Retriever consistently delivers the best results, highlighting its adaptability to various LLMs.
Figure 15: Qualitative Example - Original Abstract and Abstract Generated by Thought-Retriever. This figure presents example outputs from different methods using data from the AcademicEval-abstract-single dataset. Specifically, it shows the original abstract alongside the abstract generated by Thought-Retriever, accompanied by a comment from an expert LLM. Comparison examples generated by DPR and TF-IDF are shown in Figure 16 , while comparison examples by the long context model can be found in Figure 17 .
Figure 16: Qualitative Example - Abstracts Generated by DPR and TF-IDF. This figure presents example outputs using data from the AcademicEval-abstract-single dataset, generated by traditional methods: DPR and TF-IDF. We also include comments from an expert LLM. The original abstract and the abstract generated by our Thought-Retriever can be found in Figure 15 .
Figure 17: Qualitative Example - Abstracts Generated by Long Context Model. This figure presents example outputs using data from the AcademicEval-abstract-single dataset, generated by the long context model Nours Hermes - 32k. The original abstract and the abstract generated by our Thought-Retriever can be found in Figure 15 .
Figure 18: Arxiv Copilot Demo. This figure shows the demo built based on our proposed Thought-Retriever, which is publicly available on Hugging Face. It offers personalized academic services, aiming to test the real-world robustness of our algorithm and provide social benefits.
QA Accuracy
Reasoning Accuracy
BM25
10%
30%
TF-IDF
13%
33%
Contriever
20%
50%
DPR
17%
20%
Dragon
13%
27%
Full Context (left)
7%
20%
Appendix
Table 8: Thought-Retriever’s Effectiveness on QA and Reasoning Tasks. This table presents results from LooGLE, a recent and widely used QA and reasoning benchmark. Our Thought Retriever consistently outperforms all other baselines.
Method
F1
Contriever
0.242
Thought-Retriever using Contriever
0.290
NV-Embed
0.268
Thought-Retriever using NV-Embed
0.326
Appendix
Table 9: Ablation with latest retriever models. We substitute the Contriever component in Thought-Retriever with NV-Embed, one of the latest retriever models on MTEB, and evaluate their performance comparatively on the Abstract Single task.
Dataset
Model
F1
Win Rate
Abstract-single
RECOMP
0.237
7%
Abstract-single
Thought-Retriever
0.290
50%
Abstract-multi
RECOMP
0.202
8%
Abstract-multi
Thought-Retriever
0.275
50%
Appendix
Table 10: Comparison between Thought-Retriever and RECOMP in Abstract-single and Abstract-multi.
Task Description:
You will be shown a query and several intermediate thoughts generated by a large language model (LLM). Please evaluate each thought independently based on its usefulness, relevance, and correctness. Your rating should reflect how well the thought contributes to answering the query.
Scoring Criteria (per thought):
Score 2 (High Quality) : Relevant, helpful, logically coherent, and factually correct. Score 1 (Medium Quality) : Somewhat relevant/helpful; may have minor issues or vague content. Score 0 (Low Quality) : Irrelevant, incorrect, incoherent, or hallucinated content.
Instructions: • Evaluate each thought individually , without comparing it to others. • Focus on content quality, not writing style or fluency. • You only need to assign a score (0, 1, or 2); no explanation required.
Example:
Query: What are the key challenges in training large-scale language models?
Appendix
Table 11: Human evaluation instructions for thought quality assessment.
MemWalker
Thought-Retriever (Ours)
Type
Dataset
F1
Win Rate
Time
F1
Win Rate
Time
Academic Eval
Abs-single
0.268
40%
25.0s
0.290
50%
1.85s
Abs-multi
0.255
35%
85.0s
0.275
50%
3.20s
Rel-multi
0.212
44%
65.0s
0.216
50%
2.40s
Public
Gov Report
0.229
45%
30.0s
0.232
50%
3.50s
WCEP
0.228
39%
28.0s
0.238
50%
3.10s
Appendix
Table 12: Performance and Inference Time Comparison across Datasets. Note that MemWalker incurs high latency due to its interactive, multi-step tree navigation process.
Method
Abstract-single
Abstract-multi
Related-multi
Gov Report
WCEP
F1
Win Rate
F1
Win Rate
F1
Win Rate
F1
Win Rate
F1
Win Rate
SOTA Baseline (Qwen3)
0.245
28%
0.240
20%
0.211
35%
0.229
42%
0.235
44%
Oracle (Gold Chunks)
0.278
46%
0.255
42%
0.214
49%
0.255
62%
0.245
55%
Thought-Retriever
0.290
50%
0.275
50%
0.216
50%
0.232
50%
0.238
50%
Appendix
Table 13: Performance comparison with the SOTA baseline and the Oracle upper bound. We select Qwen3-Embed-8b as the strongest baseline. Win rate compares each method’s response with Thought-Retriever (50% represents a tie). Remarkably, Thought-Retriever achieves a >50% win rate against the Oracle on AcademicEval tasks , demonstrating that generated thoughts provide superior reasoning guidance than raw gold chunks.
HRALM (Baselines)
Thought-Retriever (Ours)
Type
Dataset
Prec.
Recall
F1
Prec.
Recall
F1
AcademicEval
Abs-single
0.82
0.65
0.72
0.80
0.88
0.84
Abs-multi
0.74
0.58
0.65
0.78
0.84
0.81
Rel-multi
0.65
0.52
0.58
0.76
0.82
0.79
Public
Gov Report
0.78
0.60
0.68
0.75
0.82
0.78
WCEP
0.71
0.62
0.66
0.77
0.83
0.80
Appendix
Table 14: Retrieval Quality Comparison: HRALM vs. Thought-Retriever. We evaluate the Chunk Coverage performance based on the precision and recall of retrieved raw data chunks. Note: While HRALM achieves slightly higher precision in single-document tasks (e.g., Gov Report) due to its extraction-based summarization, Thought-Retriever significantly outperforms it in recall and overall F1 score by bridging semantic gaps with generated thoughts.
Method
Memory Status
F1 Score on Held-out Test Set
(at start of testing)
Abstract-multi
Related-multi
Baseline (Cold Start)
Empty
0.235
0.198
Ours (Evolved)
Pre-filled (from Evolution Set)
0.250
0.210
Relative Improvement
+6.4%
+6.1%
Appendix
Table 15: Causal Evidence of Self-Evolution on Disjoint Sets. Comparison of F1 scores on the held-out Test Set (50% of data) with and without prior evolution on the disjoint Evolution Set (50% of data). The consistent improvement confirms that accumulated thoughts provide transferable benefits to novel queries.
Type
Inference Time (Seconds) ↓
Dataset
Abstract-single
Abstract-multi
Related-multi
Gov Report
WCEP
Method
Time (s)
Time (s)
Time (s)
Time (s)
Time (s)
BM25
0.42
0.45
0.48
0.75
0.62
TF-IDF
0.40
0.43
0.46
0.72
0.60
Contriever
0.65
0.72
0.68
1.10
0.95
DPR
0.62
0.70
0.66
1.05
0.92
Appendix
Table 16: Inference time comparison across different datasets. Values represent the average latency in seconds (s) per query. Lower is better. While heuristic and simple dense retrievers are the fastest, Thought-Retriever maintains a competitive inference speed compared to advanced baselines like IRCoT and long-context LLMs (Nous Hermes-32k), achieving a balance between performance and efficiency.
Large Language Model (LLM) has exhibited strong reasoning ability in text-based contexts across various domains, yet the limitation of context window poses challenges for the model on long-range inference tasks and necessitates a memory storage system. While many current storage approaches have been proposed with episodic notes and graph representations of memory, retrieval methods still primarily rely on predefined workflows or static similarity top-k over embeddings. To address this inflexibility, we introduced a novel tool-augmented autonomous memory retrieval framework (TA-Mem), which contains: (1) a memory extraction LLM agent which is prompted to adaptively chuck an input into sub-context based on semantic correlation, and extract information into structured notes, (2) a multi-indexed memory database designed for different types of query methods including both key-based lookup and similarity-based retrieval, (3) a tool-augmented memory retrieval agent which explores the memory autonomously by selecting appropriate tools provided by the database based on the user input, and decides whether to proceed to the next iteration or finalizing the response after reasoning on the fetched memories. The TA-Mem is evaluated on the LoCoMo dataset, achieving significant performance improvements over existing baseline approaches. In addition, an analysis of tool use across different question types also demonstrates the adaptivity of the proposed method.
Mengwei Yuan, Jianan Liu, Jing Yang +4
Independent Researcher Milpitas, USA, 95035 · Independent Researcher Austin, USA, 78613 · Washington University in St. Louis St. Louis, USA, 63130 +2
Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.
LLM-based autonomous agents require external memory to overcome their statelessness and limited context window for long-term interaction and dynamic knowledge reasoning. However, existing memory retrieval methods often lack adaptability and sample efficiency, and struggle to retrieve the right mixture of memories from heterogeneous stores. We propose Exploratory-Assimilating Reflection (EAR), a framework for high initial retrieval performance and sample-efficient adaptation. EAR combines two mechanisms: Exploratory Reflection, which performs iterative search to bootstrap retrieval and collect useful experiences for each query, and Assimilating Reflection, which replays these experiences from an Experience Buffer to refine a global reranker more efficiently than methods relying only on immediate rewards. Experiments show that EAR improves retrieval by up to 17.9% over the baseline retriever on two long-term dialogue benchmarks. We also show that EAR is highly sample-efficient and robust to noisy feedback.