PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agent

Prince Zizhuang Wang; Shuli Jiang

cs.AIApr 8, 2026

PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agent

Authors: Prince Zizhuang Wang, Shuli Jiang

Organizations: Carnegie Mellon University

Abstract

The development of autonomous tool-use agents for complex, long-horizon tasks in collaboration with human users has become the frontier of agentic research. During multi-turn Human-AI interactions, the dynamic and uncertain nature of user demands poses a significant challenge; agents must not only invoke tools but also iteratively refine their understanding of user intent through effective communication. While recent advances in reinforcement learning offer a path to more capable tool-use agents, existing approaches require expensive training costs and struggle with turn-level credit assignment across extended interaction horizons. To this end, we introduce PRIME (Proactive Reasoning via Iterative Memory Evolution), a gradient-free learning framework that enables continuous agent evolvement through explicit experience accumulation rather than expensive parameter optimization. PRIME distills multi-turn interaction trajectories into structured, human-readable experiences organized across three semantic zones: successful strategies, failure patterns, and user preferences. These experiences evolve through meta-level operations and guide future agent behavior via retrieval-augmented generation. Our experiments across several diverse user-centric environments demonstrate that PRIME achieves competitive performance with gradient-based methods while offering cost-efficiency and interpretability. Together, PRIME presents a practical paradigm for building proactive, collaborative agents that learn from Human-AI interaction without the computational burden of gradient-based training.

Figures & tables

Panel of Figure 1 — Figure 1: Memory library organization. Top row: Memories are organized by trajectory reward $R(\tau)$ into three semantic zones—golden (successful strategies), warning (failure patterns), and preference (user behavior patterns). Bottom row: Each memory is also annotated with its interaction stage—exploration (information gathering), verification (refining understanding), or completion (delivering solutions). Core components are color-coded: situation , action , and lesson .

Panel of Figure 2 — Figure 2: Vanilla LLM vs. PRIME on an IntentionGym task. (A) Without experience guidance, the agent jumps directly to providing solutions, makes incorrect assumptions about user preferences, and is forced to backtrack when corrected. (B) With PRIME, the agent systematically asks focused clarifying questions that efficiently cover all missing details, guided by retrieved experiences from similar past interactions. Green checks ( ✓ ) mark productive turns that address user needs; red crosses ( $\boldsymbol{\times}$ ) mark turns that assume rather than ask.

Figure 3: Comparison between base Raw models, PRIME, and RL-based models across challenging user-centric environments.

Closed-Source LLM
Model	TravelGym	TurtleGym	FunctionGym	TauGym	PersuadeGym	IntentionGym	TelepathyGym	SearchGym
GPT-4o	0.364	0.292	0.282	0.030	0.377	1.898	0.854	0.880
GPT-4o-mini	0.098	0.091	0.154	0.206	0.532	0.250	0.049	0.352
Gemini-2.5-Pro	0.347	0.274	0.410	0.194	0.425	1.590	0.902	0.928
Gemini-2.5-Flash	0.255	0.196	0.321	0.121	0.409	1.685	0.634	0.928
Open-Source LLM

Table 1: Main results on multi-turn user-interacting benchmarks. We report task completion scores across eight user-centric environments (higher is better). Gray rows indicate PRIME (ours). Blue values in parentheses show the percentage improvement over the corresponding raw open-source model.

Panel of Figure 4 — Figure 4: Efficiency and transferability.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Memory evolution operators. The library evolves through four meta-level operations applied probabilistically. Mutation sharpens vague experiences based on feedback. Generalization abstracts domain-specific successes into transferable knowledge. Crossover synthesizes complementary insights into richer experiences. Pruning removes stale entries to keep the library focused. Together, these operators enable the experience library to improve over time without gradient computation.

Panel of Figure 6 — Figure 6: Experience distillation from TurtleGym. The agent identifies the key symbolic element (snowman) in turn 1 and uses binary questions to confirm the hypothesis. Credit assignment via R2G identifies turns 1-2 as pivotal. The distilled lesson—focusing on unusual elements and using binary confirmation—generalizes to other lateral thinking puzzles.

Panel of Figure 7 — Figure 7: Experience distillation from TravelGym. The agent efficiently elicits preferences across multiple dimensions (climate, dates, budget, constraints) in 3 turns before searching. The distilled lesson emphasizes a logical hierarchy for preference discovery and the value of combining related questions.

Panel of Figure 8 — Figure 8: Experience distillation from FunctionGym. The agent uses systematic coefficient isolation to discover a linear function. The distilled lesson—unit-vector testing followed by combined verification—provides an efficient algorithm for this class of problems.

Panel of Figure 9 — Figure 9: Experience distillation from a successful IntentionGym trajectory. The agent efficiently uncovers 5 out of 6 missing details in 4 turns by strategically combining related questions. Credit assignment identifies the key turns, and the LLM distiller extracts a structured experience with applicability conditions for future contextualized retrieval. The distilled lesson — pairing related details in single questions — transfers to similar intent-discovery tasks across environments.

Panel of Figure 10 — Figure 10: Environment-specific system prompts. Each environment defines the agent’s role, available actions, and strategic guidance. At inference time, PRIME augments these base prompts with retrieved experiences from the three-zone library (golden strategies, warning patterns, user preferences) matched to the current interaction stage.

Parameter	Value	Description
Exploration
Episodes per env	100	Trajectories collected
Max turns	16	Episode horizon
Discount $\gamma$	0.8	Credit assignment decay
Distillation
Golden threshold	$R\geq 0.7$	Success classification

Appendix

Table 2: Core PRIME hyperparameters.

Parameter	Value	Description
Evolution iterations $N$	5	Meta-optimization cycles
Mutation prob $p_{\text{mut}}$	0.10	Per-experience rate
Generalization prob $p_{\text{gen}}$	0.05	Abstraction rate
Crossover prob $p_{\text{cross}}$	0.02	Combination rate
Generalization threshold	0.7	Min success rate
Pruning interval	2 iterations	Removal frequency

Appendix

Table 3: Evolution phase hyperparameters.

Panel of Figure 11 — Figure 11: Experience-guided inference via contextualized retrieval. At each turn, PRIME determines the interaction stage, prefilters candidate experiences by environment and stage, uses LLM reasoning to select the most applicable experiences, and augments the agent prompt with zone-organized guidance. The bottom panel shows an example retrieval at turn 2 of an IntentionGym episode, where golden, warning, and preference experiences each contribute different types of guidance.

Self-Improving Agents

Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.

See all 169 papers

PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents
Jun 15, 2026Zhenbang Du, Jun Luo, Zhiwei Zheng +8Learning from Demonstration LLM Agent Training
Georgia Institute of Technology · Ohio State University · University of Pennsylvania +1
EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents
Oct 12, 2026Min Cai, Dan Zhang, Gao Xing +3LLM Agent Training RL for LLM Agents
University of Alberta · National University of Singapore · Kuaishou Technology +2
InfoPO: Information-Driven Policy Optimization for User-Centric Agents
Feb 28, 2026Fanqi Kong, Jiayi Zhang, Mingyi Deng +3LLM Agent Training RL for LLM Agents
Peking University · DeepWisdom · The Hong Kong University of Science and Technology (Guangzhou) +1

Jun 15, 2026J/K move · Enter open · S save

PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

Zhenbang Du, Jun Luo, Zhiwei Zheng, Xiangchi Yuan+7

Georgia Institute of Technology · Ohio State University · University of Pennsylvania+1

Learning from Demonstration LLM Agent Training RL Post-Training Tool Use RL for Tool Use

Open paper AskPDF arXiv

Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that provides annealed process supervision over reasoning and tool-call components. To reduce over-reliance on the training-only trace context, PACT incorporates prompt-only anchoring with standard prompt-only RL updates. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently outperforms strong SFT- and RL-based baselines. Further analyses support the complementarity of the two training objectives and the benefits of component-aware supervision and prompt-only anchoring. These results highlight the effectiveness of privileged trace co-training for multi-turn tool-use learning. The project page is available at https://zhenbangdu.github.io/pact-project-page.

Abstract

Self-Improving Agents

Explore similar work

PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

InfoPO: Information-Driven Policy Optimization for User-Centric Agents