Beyond Outcome Feedback: Trajectory-Quality-Guided Experience-based Learning

Bingjie Zhang; Yibo Yang; Qiantong Liu; Benshuo Fu; Yi Chang; Dandan Guo

cs.AIOct 10, 2026

Beyond Outcome Feedback: Trajectory-Quality-Guided Experience-based Learning

Authors: Bingjie Zhang, Yibo Yang, Qiantong Liu, Benshuo Fu, Yi Chang, Dandan Guo

Organizations: School of Artificial Intelligence, Jilin University · School of Artificial Intelligence, Shanghai Jiao Tong University · King Abdullah University of Science and Technology

Abstract

Experience-based learning enables large language model (LLM) agents to improve over time by reusing knowledge from past executions. However, existing methods typically rely on final task outcomes as feedback for experience learning, overlooking the execution process and limiting their ability to evaluate and optimize experience based on trajectory quality. To address this limitation, we propose TQG-Exp, a trajectory-quality-guided experience-based learning framework that introduces process feedback. Specifically, we introduce Trajectory Quality Loss (TQL), which jointly considers task completion and process quality to provide a learning signal beyond final task outcomes. Based on this signal, we employ paired rollouts with and without retrieved experience to guide experience optimization. Representative trajectories are selected based on their outcomes and trajectory quality to extract task-level and task-type-level experiences. Meanwhile, differences in trajectory quality between the paired groups are used to estimate the contribution of retrieved experience and update its utility. The learned utility is then combined with semantic relevance to guide subsequent experience retrieval. Experiments on ALFWorld, WebShop, and ScienceWorld with two backbone LLMs show that TQG-Exp outperforms existing experience-based methods, improving Avg.@3 over the strongest baseline by 4.50 and 8.55 percentage points on DeepSeek-V4-Flash and Qwen3.5-Flash, respectively.

Figures & tables

Figure 1: Illustration of outcome-level and process-level signal in reinforcement learning (top) and experience-based learning (bottom).

Figure 2: Overview of TQG-Exp. The top illustrates the overall experience learning loop, while the bottom panels detail process-aware trajectory evaluation, hierarchical experience abstraction, and counterfactual experience credit assignment with utility updates.

Model	Method	ALFWorld				WebShop		ScienceWorld		Avg.
		Seen		Unseen		WebShop		ScienceWorld		Avg.
		Avg.@3 $\uparrow$	Pass@3 $\uparrow$	Avg.@3 $\uparrow$	Pass@3 $\uparrow$	Avg.@3 $\uparrow$	Pass@3 $\uparrow$	Avg.@3 $\uparrow$	Pass@3 $\uparrow$	Avg.@3 $\uparrow$	Pass@3 $\uparrow$
DeepSeek-V4-Flash	No Experience	68.57	76.43	74.88	85.82	42.33	52.00	47.00	60.00	58.20	68.56
	ReasoningBank ( Ouyang et al., 2025 )	75.24	82.14	75.37	85.07	39.67	48.00	48.33	55.00	59.65	67.55
	MemP ( Fang et al., 2025 )	84.76	91.43	88.06	94.03	36.33	48.00	57.33	63.00	66.62	74.12
	MemRL ( Zhang et al., 2026 )	90.00	92.86	90.30	93.28	73.67	78.00	69.33	76.00	80.83	85.04

Table 1: Performance comparison across ALFWorld, WebShop, and ScienceWorld. Best results are in bold , and second-best results are underlined .

Split	Method	Performance & Efficiency
Split	Method	Avg.@3 $\uparrow$	Avg. Steps $\downarrow$	Tok./Ep. $\downarrow$	Tok./Succ. $\downarrow$
Seen	No Experience	68.57	18.17	27.11k	40.10k
	MemRL	90.00	11.13	114.34k	127.04k
	TQG-Exp	96.19	10.27	38.06k	39.57k
Unseen	No Experience	74.88	18.19	27.89k	37.25k
	MemRL	90.30	12.31	125.28k	138.73k

Table 2: Performance and efficiency on ALFWorld with DeepSeek-V4-Flash.

Model	Components				ALFWorld
Model	Hier. Exp.	Util. Ret.	TQL	Traj. Ret.	Seen	Unseen
DeepSeek-V4-Flash	✓	✓	✓	✓	96.19	90.80
	✓	✓	✓	✗	91.43	89.80
	✓	✓	✗	✗	78.81	79.35
	✓	✗	✗	✗	84.76	83.08
	✗	✓	✓	✓	93.57	89.55

Table 3: Ablation on ALFWorld seen and unseen tasks. All results are Avg.@3. Hier. Exp., Util. Ret., and Traj. Ret. denote Hierarchical Experience, Utility-Aware Experience Retrieval, and Trajectory Retrieval.

Figure 3: Sensitivity of performance to

\alpha

on ALFWorld using DeepSeek-V4-Flash.

Figure 4: Epoch-wise TQL sum and task success rate during experience learning. Solid lines indicate TQL sums (left axis), while dashed lines indicate task success rates (right axis).

Figure 5: Semantic evolution of the ALFWorld experience bank. Circles denote task experiences, triangles denote task-type experiences, color indicates utility, and marker size indicates retrieval count.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Method	Average@3 $\uparrow$	Avg. Steps $\downarrow$	Token Efficiency
Split	Method	Average@3 $\uparrow$	Avg. Steps $\downarrow$	Tokens / Episode $\downarrow$	Tokens / Success $\downarrow$
Webshop	No Experience	42.33	21.59	47.20k	111.49k
	MemRL	73.67	12.60	205.00k	278.28k
	TQG-Exp	79.67	8.77	55.90k	70.16k
Scienceworld	No Experience	47.00	18.77	30.46k	64.80k
	MemRL	69.33	14.41	220.61k	318.18k

Appendix

Table 4: Task Performance and Interaction Efficiency across Benchmarks using DeepSeek-V4-Flash.

Panel of Figure 6 — Figure 6: Evolution of the epoch-level TQL sum on WebShop and ScienceWorld during training with DeepSeek-V4-Flash.

Figure 7: Sizes of the task-level and task-type-level experience banks on ALFWorld using DeepSeek-V4-Flash.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 590 papers

ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents
May 29, 2026Tao Feng, Chongrui Ye, Tianyang Luo +8Continual Learning for LLM Agents LLM Agent Skill Learning
University of Illinois Urbana-Champaign · 2Nanyang Technological University · 3Meta Monetization AI
Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learning
Jun 23, 2026Shiding Zhu, Yudi Qi, Yajie Wang +6LLM Agents Self-Evolving Agents
Zhejiang University · Shenzhen International Graduate School, Tsinghua University · Independent Researcher +3
Retrieval-Augmented LLM Agents: Learning to Learn from Experience
Mar 18, 2026Thomas Palmeira Ferraz, Romain Deffayet, Vassilina Nikoulina +2Fine-Tuning LLM Agent Training
NAVER LABS Europe, Meylan, France

May 29, 2026J/K move · Enter open · S save

ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents

Tao Feng, Chongrui Ye, Tianyang Luo, Jingjun Xu+7

University of Illinois Urbana-Champaign · 2Nanyang Technological University · 3Meta Monetization AI

Continual Learning for LLM Agents LLM Agent Skill Learning Graph-Based Agent Memory Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents have shown strong capabilities in reasoning, tool use, and multi-step interaction, but they often solve tasks from scratch and fail to reuse successful strategies or failure lessons from prior experience. Fine-tuning on collected experience can improve reuse, but it is inflexible when stronger or more suitable executors emerge. We propose ExpGraph, a model-agnostic experience learning framework that enables frozen and replaceable LLM executors to improve through external experience reuse without parameter updates. ExpGraph summarizes historical trajectories into reusable skills and failure lessons, organizes them as nodes in a self-evolving experience graph, and retrieves useful experiences through graph diffusion and utility-aware ranking. A lightweight retrieval copilot is trained with reinforcement learning using feedback that compares executor performance with and without retrieved experiences, while the graph is updated online from downstream task outcomes. We evaluate ExpGraph on ExpSuite, covering question answering, mathematical reasoning, code generation, and multi-step agentic environments including ALFWorld and AppWorld. ExpGraph improves over the strongest baseline by 12.2% and 4.7% on static tasks with smaller and larger executors, and by 21.4% and 12.7% in agentic environments, while reducing average interaction steps by 12.7% and 21.6%. Ablations show that graph-structured experience, utility-aware ranking, and adaptive retrieval jointly enable effective experience reuse across diverse tasks and executor models.

Abstract

LLM Agent Training

Explore similar work

ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents

Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learning

Retrieval-Augmented LLM Agents: Learning to Learn from Experience