Learning to Accumulate Knowledge with Mutual Information

Yuyang Zhao; Lizi Liao; Leyang Shen; Xiaoyan Zhao; Yang Zhang; Fuli Feng; Xiangnan He

cs.AIOct 7, 2026

Learning to Accumulate Knowledge with Mutual Information

Authors: Yuyang Zhao, Lizi Liao, Leyang Shen, Xiaoyan Zhao, Yang Zhang, Fuli Feng, Xiangnan He

Organizations: University of Science and Technology of China · Singapore Management University · National University of Singapore

Abstract

Large language model (LLM) agents can improve their performance by reusing knowledge distilled from past interactions. However, curating new experiences into a knowledge bank that becomes more useful as it grows remains challenging. Effective knowledge accumulation should limit redundant overlap among entries and ensure that new knowledge contributes beyond what the bank already provides. Yet training a curator with Group Relative Policy Optimization (GRPO) on standalone task success can reinforce general guidance even when it duplicates existing knowledge. Therefore, we propose Knowledge Weaver, a reinforcement learning framework that trains a language model to curate reusable knowledge from agent trajectories. We couple feedback inspired by token-wise mutual information (MI) with marginal success rewards to guide knowledge accumulation. Together, these signals encourage the curator to preserve distinct information from experience and produce entries that improve task success when added to existing knowledge. Standalone success rewards also favor entries that are useful on their own. On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates of 54.0% and 42.0% with k=10 retrieved entries, exceeding GRPO by 16.9 and 18.7 percentage points, respectively. Its knowledge banks also outperform the evaluated prompt-based and established banks, including human-written banks, in overall ALFWorld success rate and WebShop score with the executor frozen. Our codebase is available at https://github.com/LaoKuiZe/Knowledge-Weaver.

Figures & tables

Figure 1: Conceptual illustration of general guidance and complementary knowledge. General guidance can remain useful for individual tasks while providing limited additional information across entries. Knowledge Weaver combines MI-inspired feedback with execution rewards to encourage experience-dependent knowledge and incremental utility.

Figure 2: Overview of Knowledge Weaver. Token-level MI feedback promotes experience-dependent knowledge, while execution rewards assess individual utility and marginal gains over retrieved knowledge. Only the curator is trained and the executor remains frozen.

	ALFWorld (SR%)							WebShop
Method	Pick	Clean	Heat	Cool	Look	Pick Two	Overall	Score	SR%
Zero-shot	$30.6$	$10.8$	$31.9$	$30.2$	$24.1$	$25.5$	$24.6_{\pm 1.5}$	$53.5_{\pm 1.9}$	$27.3_{\pm 1.5}$

Table 1: Performance of knowledge banks with a frozen Qwen3.5-4B executor on ALFWorld and WebShop. Each bank supplies

k=10

retrieved entries per task and zero-shot execution uses no external knowledge. We report success rate (SR, %) and mean environment score on WebShop. All results are averaged over 3 seeds, with standard deviations shown as subscripts for the Overall and WebShop columns. Bold and underlining mark the best and second-best in each column.

Explore similar work

ExpWeaver: LLM Agents Learn from Experience via Latent RAG
May 31, 2026Tao Feng, Tianyang Luo, Jingjun Xu +5Large Language Model Agents Reusable Knowledge
Department of Computer Science, University of Illinois Urbana-Champaign, Urbana, IL, USA · Meta Monetization AI, USA
Retrieval-Augmented LLM Agents: Learning to Learn from Experience
Mar 18, 2026Thomas Palmeira Ferraz, Romain Deffayet, Vassilina Nikoulina +2Large Language Model Agents Memory-Augmented Framework
NAVER LABS Europe, Meylan, France
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Jul 4, 2026Weiyang Guo, Zesheng Shi, Longhui Zhang +3Retrying Offline Reinforcement Learning
Harbin Institute of Technology, Shenzhen, China

May 31, 2026, cs.CLJ/K move · Enter open · S save

ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Tao Feng, Tianyang Luo, Jingjun Xu, Zhigang Hua+4

Department of Computer Science, University of Illinois Urbana-Champaign, Urbana, IL, USA · Meta Monetization AI, USA

Large Language Model Agents Reusable Knowledge Weave

Open paper AskPDF arXiv

Experience learning has achieved promising results in enhancing LLM agent planning and reasoning by integrating past interactions as reusable knowledge. However, existing methods remain confined to explicit text space, retrieving experiences via semantic similarity and concatenating them into the context window, leading to substantial token overhead and a decoupled architecture that separates retrieval from generation. To address these limitations, we propose ExpWeaver, a framework that enables LLM agents to learn from experience via latent retrieval-augmented generation, without requiring a separate RAG module. ExpWeaver encodes experiences using the LLM's own hidden states, retrieves relevant experiences directly in latent space at each decoding step, and integrates them through cross-attention aggregation and gated residual mechanisms. The entire pipeline is optimized end-to-end with reinforcement learning, supporting both generative and ranking tasks. We evaluate ExpWeaver on 13 diverse tasks spanning question answering, reasoning, coding, scientific prediction, and recommendation. Results demonstrate that ExpWeaver achieves state-of-the-art performance on 12 out of 13 tasks, outperforming the strongest baseline by over 6.8%; maintains token efficiency comparable to non-retrieval baselines while text-based retrieval methods require 1.5 to 2 times more tokens; and exhibits superior cross-domain generalization, outperforming the strongest baseline by 16.32% under zero-shot transfer and 15.21% under few-shot transfer. Our code for ExpWeaver is released at https://github.com/ulab-uiuc/ExpWeaver.

Abstract

Explore similar work

ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Retrieval-Augmented LLM Agents: Learning to Learn from Experience

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry