cs.CLApr 14, 2026
SaveThought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems
Organizations: University of Illinois Urbana-Champaign · Carnegie Mellon University
Abstract
Large language models (LLMs) have transformed AI research thanks to their powerful internal capabilities and knowledge. However, existing LLMs still fail to effectively incorporate the massive external knowledge when interacting with the world. Although retrieval-augmented LLMs are proposed to mitigate the issue, they are still fundamentally constrained by the context length of LLMs, as they can only retrieve top-K raw data chunks from the external knowledge base which often consists of millions of data chunks. Here we propose Thought-Retriever, a novel model-agnostic algorithm that helps LLMs generate output conditioned on arbitrarily long external data, without being constrained by the context length or number of retrieved data chunks. Our key insight is to let an LLM fully leverage its intermediate responses generated when solving past user queries (thoughts), filtering meaningless and redundant thoughts, organizing them in thought memory, and retrieving the relevant thoughts when addressing new queries. This effectively equips LLM-based agents with a self-evolving long-term memory that grows more capable through continuous interaction. Besides algorithmic innovation, we further meticulously prepare a novel benchmark, AcademicEval, which requires an LLM to faithfully leverage ultra-long context to answer queries based on real-world academic papers. Extensive experiments on AcademicEval and two other public datasets validate that Thought-Retriever remarkably outperforms state-of-the-art baselines, achieving an average increase of at least 7.6% in F1 score and 16% in win rate across various tasks. More importantly, we further demonstrate two exciting findings: (1) Thought-Retriever can indeed help LLM self-evolve after solving more user queries; (2) Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.
Figures & tables
Figure 1: Why Thought-Retriever helps . (a) A standard RALM is limited by the number of retrieved chunks. The retrieved data fails to cover all the necessary data chunks (red chunks) for a user query. (b) A hierarchical RALM retrieves summaries , generated independently from user queries, which could improve recall at the cost of lower precision. (c) Thought-Retriever leverages past LLM thoughts collected from answering user queries, with little computational overhead. Thought-Retriever balances low-level facts and high-level thoughts, leading to high precision and recall.
Figure 2: Thought-Retriever Framework . (a) Thought retrieval: Upon receiving a user query, Thought-Retriever retrieves top-K data chunks from the mixture of external knowledge and thought memory based on embedding similarity; (b) Answer and confidence generation: The LLM generates the answer for the user query based on the retrieved data chunks; (c) Thought generation: The LLM further generates thoughts and its confidence based on the user query and the generated answer; (d) Thought merge: The calculation of similarity is used to measure whether the generated thought will cause redundancy in data chunks; (e) Thought memory update: Meaningless and redundant thoughts are removed and the remaining novel thoughts are used to update the thought memory.
Figure 3: Thought and Confidence Generation Prompt. This prompt is used for Thought and Confidence Generation as described in Section 2.3 . It evaluates whether the answer is valid and meaningful, and then summarizes the query and answer into a thought.
| Dataset | Task Type | Avg. len | Cases |
| AcademicEval | |||
| Abstract-single | Single Sum | 8,295 | 100 |
| Abstract-multi | Multi Sum | 33,637 | 30 |
| Related-multi | Multi Related | 22,107 | 30 |
| Public Datasets | |||
| Gov Report | Single QA | 8,910 | 100 |
Table 1: Overview of Datasets: task types, average length, and number of cases.
| Type | AcademicEval | Public | ||||||||
| Dataset | Abstract-single | Abstract-multi | Related-multi | Gov Report | WCEP | |||||
| Method | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate |
| BM25 | 0.212 | 7% | 0.232 | 7% | 0.203 | 40% | 0.211 | 30% | 0.178 | 31% |
| TF-IDF | 0.202 | 4% | 0.225 | 4% | 0.207 | 40% | 0.195 | 35% | 0.223 | 34% |
| Contriever | 0.242 | 13% | 0.232 | 15% | 0.201 | 35% | 0.223 | 40% | 0.211 | 40% |
| DPR | 0.206 | 4% | 0.226 | 4% | 0.196 | 30% | 0.188 | 20% | 0.201 | 33% |
Table 2: Thought-Retriever consistently outperforms all the baselines in fact retrieval datasets . Bold and underline denote the best and second-best results. F1 score evaluates the similarity with the ground truth, higher is better. Win Rate represents the frequency with which a method’s response is preferred over Thought-Retriever by the evaluator. The 50% entry for Thought-Retriever serves as the reference point (a tie); consequently, for other baselines, a win rate lower than 50% indicates that they are outperformed by Thought-Retriever. Note that the maximum context length is 2,000 tokens for all retriever-based methods and Thought-Retriever employs Contriever as its retriever.
| Setting | Retriever-origin | Response-direct | Retriever-other | Thought-Retriever |
| F1 | 0.25 | 0.19 | 0.22 | 0.24 |
Table 3: Thought-Retriever can help the LLM quickly learn from other LLMs . Retriever-origin is the golden setting that retrieves original facts, others are comparative settings without original facts.
Figure 4: Thoughts from other LLMs help respond without fact. It presents an illustrative example in which our LLM communicates with four other LLMs, each an expert in a different field. These expert LLMs are assigned specific roles (e.g., doctor) with different background knowledge. Our LLM is then able to rapidly learn from their thoughts and incorporate them as external knowledge.
Figure 5: Deeper thoughts help abstract queries. This figure illustrates the correlation between six questions, categorized by their level of abstraction as evaluated by expert LLM (x-axis), and the abstraction level of the corresponding retrieved information (y-axis). The questions are grouped into three categories: high abstraction (top 2 questions), medium abstraction, and low abstraction. Keywords from each question are displayed next to their corresponding data points for clarity.
Figure 6: Thought-Retriever can indeed help LLM self-evolve after solving more user queries. It illustrates that the performance of LLM across two datasets shows an upward trend as the number of thoughts increases.
Figure 7: Contriever and thoughts filtering are suitable for Thought-Retriever. Ablation study of 6 methods on two datasets helps us decide on important components of Thought-Retriever.
Figure 8: Thought-Retriever performs better in balancing recall and precision. The dotted line indicates the exact balance between precision and recall. The closer the dotted line is, the better the balance is.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Attribute | Description | |
| Abstract-Single | ‘title’ | The title of the academic paper. |
| ‘abstract’ as label | The abstract of the academic paper. | |
| ‘main_content’ | The content of the paper excluding the abstract and the conclusion. | |
| Abstract-Multi | ‘title 1’ | The title of the first academic paper. |
| ‘abstract 1’ | The abstract of the first academic paper. | |
| ‘main_content 1’ | The content of the first paper excluding the abstract and the conclusion. |
Table 4: AcademicEval Dataset Documentation. This table presents the specific format of the data in our AcademicEval dataset.
Figure 9: AcademicEval Usage Instructions. This figure provides a visualization of the usage instructions for the AcademicEval dataset, as described in Section A.2 , to aid understanding.
Figure 10: Prompt for Writing Abstracts. This prompt was used in our experiment to ask the LLM to write an abstract based on the retrieved information. We provided in-context instructions to guide the LLM in producing higher-quality responses.
Figure 11: Abstract Multi Ground Truth Prompt. This prompt was used in our experiment on the Academic-abstract-multi dataset. Specifically, for each data entry, we summarize the abstracts of five papers in the entry to create the ground truth. To ensure high-quality generation, we utilized GPT-4o as the expert LLM to synthesize these summaries based on the provided prompt.
Figure 12: Prompt for Writing Related Works. This prompt was used in our experiment to ask the LLM to write a related work section based on the original paper’s abstract and the retrieved related materials. We also provided an example of in-context learning to enable the LLM to perform more effectively on this challenging task.
Figure 13: User Query Formation Prompt. This figure presents the prompt used to model real-world user queries. Specifically, it includes two methods: template-based query formation, where general question templates are created to be suitable for a wide range of papers, and LLM-based query formation, where this prompt is used to ask an LLM to generate diverse queries.
Figure 14: AI Evaluation Prompt. This prompt is used for the AI Evaluation metric Win Rate, as described in Section 4 . Given two generated answers and the ground truth answer, we ask the expert LLM to determine which generated answer aligns more closely with the ground truth.
| Query | What has driven significant progress in various NLP tasks in recent years? |
| Response | According to the text, significant progress in various NLP tasks in recent years has been driven in part by benchmarks such as GLUE, whose leaderboards rank models by how well they perform on these diverse tasks. |
| Prompt template | Input: Given query:{query}, given response:{response}. Based on the provided query and its corresponding response, perform the following step: succinctly summarize both the question and answer into a coherent knowledge point, forming a fluent passage. |
| Thought candidate | Here is a summarized knowledge point: In recent years, significant progress has been made in various Natural Language Processing (NLP) tasks. A key driver of this progress is the development of benchmarks, such as GLUE, which provide a standardized way to evaluate and compare the performance of different models on a range of diverse NLP tasks. These benchmarks, which often take the form of leaderboards, rank models based on their performance, fostering competition and innovation in the field. As a result, researchers and developers have been motivated to improve their models, leading to significant advancements in NLP capabilities. |
Table 5: Example of thought construction . For a given query, Thought-Retriever retrieves the corresponding data chunks (thoughts and original data chunks) from the knowledge corpus to get the response. Then, Thought-Retriever integrates the query and response into a prompt through the prompt template and obtains the final thought candidate through the output of LLM. This thought candidate is then evaluated to determine whether it is correct and not redundant.
| Abstraction | Rank (expert LLM) | Query |
| High | 6 (Most Abstract) | "What are the broader future implications of user-centric utility in NLP model evaluation?" |
| High | 5 | “Please craft an abstract summarizing the key points from the provided text.” |
| Medium | 4 | "What are some of the limitations of this study?" |
| Medium | 3 | “What are the key methods introduced in this paper?” |
| Low | 2 | "Please explain the term Minerva to me." |
| Low | 1 (Least Abstract) | "How many benchmarks are used to test the model’s long context understanding ability in this paper?" |
Table 6: Sample Queries Used in Abstraction Level Case Study. This table presents sample queries from the case study conducted in Section 4 , which demonstrates how Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.
| Type | Abstract-single | Abstract-multi | ||||||
| LLM | Qwen-7b | Llama-3-70b | Qwen-7b | Llama-3-70b | ||||
| Method | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate |
| BM25 | 0.196 | 3% | 0.22 | 13% | 0.232 | 7% | 0.233 | 14% |
| TF-IDF | 0.192 | 3% | 0.21 | 17% | 0.220 | 3% | 0.224 | 12% |
| Contriever | 0.231 | 10% | 0.238 | 18% | 0.229 | 4% | 0.228 | 19% |
| DPR | 0.209 | 4% | 0.215 | 18% | 0.222 | 3% | 0.222 | 11% |
Table 7: Thought-Retriever demonstrates adaptability across different LLMs . This table compares Thought-Retriever’s performance against baselines on Abstract-single and Abstract-multi tasks using Qwen-7B and Llama-3-70B models. Thought-Retriever consistently delivers the best results, highlighting its adaptability to various LLMs.
Figure 15: Qualitative Example - Original Abstract and Abstract Generated by Thought-Retriever. This figure presents example outputs from different methods using data from the AcademicEval-abstract-single dataset. Specifically, it shows the original abstract alongside the abstract generated by Thought-Retriever, accompanied by a comment from an expert LLM. Comparison examples generated by DPR and TF-IDF are shown in Figure 16 , while comparison examples by the long context model can be found in Figure 17 .
Figure 16: Qualitative Example - Abstracts Generated by DPR and TF-IDF. This figure presents example outputs using data from the AcademicEval-abstract-single dataset, generated by traditional methods: DPR and TF-IDF. We also include comments from an expert LLM. The original abstract and the abstract generated by our Thought-Retriever can be found in Figure 15 .
Figure 17: Qualitative Example - Abstracts Generated by Long Context Model. This figure presents example outputs using data from the AcademicEval-abstract-single dataset, generated by the long context model Nours Hermes - 32k. The original abstract and the abstract generated by our Thought-Retriever can be found in Figure 15 .
Figure 18: Arxiv Copilot Demo. This figure shows the demo built based on our proposed Thought-Retriever, which is publicly available on Hugging Face. It offers personalized academic services, aiming to test the real-world robustness of our algorithm and provide social benefits.
| QA Accuracy | Reasoning Accuracy | |
| BM25 | 10% | 30% |
| TF-IDF | 13% | 33% |
| Contriever | 20% | 50% |
| DPR | 17% | 20% |
| Dragon | 13% | 27% |
| Full Context (left) | 7% | 20% |
Table 8: Thought-Retriever’s Effectiveness on QA and Reasoning Tasks. This table presents results from LooGLE, a recent and widely used QA and reasoning benchmark. Our Thought Retriever consistently outperforms all other baselines.
| Method | F1 |
| Contriever | 0.242 |
| Thought-Retriever using Contriever | 0.290 |
| NV-Embed | 0.268 |
| Thought-Retriever using NV-Embed | 0.326 |
Table 9: Ablation with latest retriever models. We substitute the Contriever component in Thought-Retriever with NV-Embed, one of the latest retriever models on MTEB, and evaluate their performance comparatively on the Abstract Single task.
| Dataset | Model | F1 | Win Rate |
| Abstract-single | RECOMP | 0.237 | 7% |
| Abstract-single | Thought-Retriever | 0.290 | 50% |
| Abstract-multi | RECOMP | 0.202 | 8% |
| Abstract-multi | Thought-Retriever | 0.275 | 50% |
Table 10: Comparison between Thought-Retriever and RECOMP in Abstract-single and Abstract-multi.
| Task Description: |
| You will be shown a query and several intermediate thoughts generated by a large language model (LLM). Please evaluate each thought independently based on its usefulness, relevance, and correctness. Your rating should reflect how well the thought contributes to answering the query. |
| Scoring Criteria (per thought): |
| Score 2 (High Quality) : Relevant, helpful, logically coherent, and factually correct. Score 1 (Medium Quality) : Somewhat relevant/helpful; may have minor issues or vague content. Score 0 (Low Quality) : Irrelevant, incorrect, incoherent, or hallucinated content. |
| Instructions: • Evaluate each thought individually , without comparing it to others. • Focus on content quality, not writing style or fluency. • You only need to assign a score (0, 1, or 2); no explanation required. |
| Example: |
| Query: What are the key challenges in training large-scale language models? |
Table 11: Human evaluation instructions for thought quality assessment.
| MemWalker | Thought-Retriever (Ours) | ||||||
| Type | Dataset | F1 | Win Rate | Time | F1 | Win Rate | Time |
| Academic Eval | Abs-single | 0.268 | 40% | 25.0s | 0.290 | 50% | 1.85s |
| Abs-multi | 0.255 | 35% | 85.0s | 0.275 | 50% | 3.20s | |
| Rel-multi | 0.212 | 44% | 65.0s | 0.216 | 50% | 2.40s | |
| Public | Gov Report | 0.229 | 45% | 30.0s | 0.232 | 50% | 3.50s |
| WCEP | 0.228 | 39% | 28.0s | 0.238 | 50% | 3.10s | |
Table 12: Performance and Inference Time Comparison across Datasets. Note that MemWalker incurs high latency due to its interactive, multi-step tree navigation process.
| Method | Abstract-single | Abstract-multi | Related-multi | Gov Report | WCEP | |||||
| F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | F1 | Win Rate | |
| SOTA Baseline (Qwen3) | 0.245 | 28% | 0.240 | 20% | 0.211 | 35% | 0.229 | 42% | 0.235 | 44% |
| Oracle (Gold Chunks) | 0.278 | 46% | 0.255 | 42% | 0.214 | 49% | 0.255 | 62% | 0.245 | 55% |
| Thought-Retriever | 0.290 | 50% | 0.275 | 50% | 0.216 | 50% | 0.232 | 50% | 0.238 | 50% |
Table 13: Performance comparison with the SOTA baseline and the Oracle upper bound. We select Qwen3-Embed-8b as the strongest baseline. Win rate compares each method’s response with Thought-Retriever (50% represents a tie). Remarkably, Thought-Retriever achieves a >50% win rate against the Oracle on AcademicEval tasks , demonstrating that generated thoughts provide superior reasoning guidance than raw gold chunks.
| HRALM (Baselines) | Thought-Retriever (Ours) | ||||||
| Type | Dataset | Prec. | Recall | F1 | Prec. | Recall | F1 |
| AcademicEval | Abs-single | 0.82 | 0.65 | 0.72 | 0.80 | 0.88 | 0.84 |
| Abs-multi | 0.74 | 0.58 | 0.65 | 0.78 | 0.84 | 0.81 | |
| Rel-multi | 0.65 | 0.52 | 0.58 | 0.76 | 0.82 | 0.79 | |
| Public | Gov Report | 0.78 | 0.60 | 0.68 | 0.75 | 0.82 | 0.78 |
| WCEP | 0.71 | 0.62 | 0.66 | 0.77 | 0.83 | 0.80 | |
Table 14: Retrieval Quality Comparison: HRALM vs. Thought-Retriever. We evaluate the Chunk Coverage performance based on the precision and recall of retrieved raw data chunks. Note: While HRALM achieves slightly higher precision in single-document tasks (e.g., Gov Report) due to its extraction-based summarization, Thought-Retriever significantly outperforms it in recall and overall F1 score by bridging semantic gaps with generated thoughts.
| Method | Memory Status | F1 Score on Held-out Test Set | |
| (at start of testing) | Abstract-multi | Related-multi | |
| Baseline (Cold Start) | Empty | 0.235 | 0.198 |
| Ours (Evolved) | Pre-filled (from Evolution Set) | 0.250 | 0.210 |
| Relative Improvement | +6.4% | +6.1% | |
Table 15: Causal Evidence of Self-Evolution on Disjoint Sets. Comparison of F1 scores on the held-out Test Set (50% of data) with and without prior evolution on the disjoint Evolution Set (50% of data). The consistent improvement confirms that accumulated thoughts provide transferable benefits to novel queries.
| Type | Inference Time (Seconds) | ||||
| Dataset | Abstract-single | Abstract-multi | Related-multi | Gov Report | WCEP |
| Method | Time (s) | Time (s) | Time (s) | Time (s) | Time (s) |
| BM25 | 0.42 | 0.45 | 0.48 | 0.75 | 0.62 |
| TF-IDF | 0.40 | 0.43 | 0.46 | 0.72 | 0.60 |
| Contriever | 0.65 | 0.72 | 0.68 | 1.10 | 0.95 |
| DPR | 0.62 | 0.70 | 0.66 | 1.05 | 0.92 |
Table 16: Inference time comparison across different datasets. Values represent the average latency in seconds (s) per query. Lower is better. While heuristic and simple dense retrievers are the fastest, Thought-Retriever maintains a competitive inference speed compared to advanced baselines like IRCoT and long-context LLMs (Nous Hermes-32k), achieving a balance between performance and efficiency.
More in this topic
Memory-Augmented Language Model Agents
Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.
See all 443 papers