AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories

Zhiyi Lyu; Yewen Li; Longtao Zheng; Shengtian Yang; Lang Feng; Lei Feng; Peng Jiang; Kun Gai

cs.AISep 5, 2026

AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories

Authors: Zhiyi Lyu, Yewen Li, Longtao Zheng, Shengtian Yang, Lang Feng, Lei Feng, Peng Jiang, Kun Gai,

Organizations: Nanyang Technological University · Kuaishou Technology · Southeast University

Abstract

LLM-based agents are increasingly deployed in real-world applications through tool-use APIs, yet training them for specific environments remains fundamentally difficult: real-world applications provide no pre-defined tasks or verifiers, no faithful simulators, and limited budget for large-scale environment interaction. In this paper, we propose AgentBrew, an offline training framework that learns effective tool-use policies from a single batch of raw interaction trajectories, without task verifiers or iterative on-policy rollouts. The agent first explores the target environment to collect a raw trajectory corpus without quality filtering. To extract training signal from this noisy corpus, retrospective task inference reconstructs an aligned instruction for each trajectory based on its actual outcome, and PMI-Based credit assignment decomposes the trajectory's total information about the inferred instruction into additive per-action credits via pointwise mutual information (PMI). These credits weight the policy training objective, amplifying informative actions while suppressing ineffective ones. On three real-world MCP applications (GitHub, Notion, PostgreSQL), AgentBrew improves Qwen3-32B by +8.7 Acc / +9.7 Score on average, surpassing Qwen3-235B (+2.3 / +4.4) and outperforming rejection sampling (+5.9 / +10.3). These results demonstrate that fine-grained offline learning can recover useful supervision from raw trajectories that filtering-based approaches would discard. The code is available at https://github.com/alphatogo/AgentBrew

Figures & tables

Figure 1: Left: trajectory-level filtering discards entire trajectories that fail the original task, losing potentially useful actions. Right: AgentBrew infers a better-aligned task for each trajectory and assigns per-action credits under the inferred task, enabling the entire corpus to contribute to training.

Figure 2: Overview of AgentBrew. (1) The agent explores $\mathcal{E}$ to propose grounded tasks and collects raw trajectories. (2) Retrospective task inference recovers aligned instructions $\hat{I}_{n}$ , and PMI-based credit assignment scores each action. (3) Normalized credits weight the policy training objective.

Method	GitHub		Notion		Postgres		Average
Method	Acc.	Score	Acc.	Score	Acc.	Score	Acc.	Score
Frontier Proprietary Models
GPT-5	64.2	85.4	42.9	83.4	42.9	61.7	50.0	76.8
Claude Sonnet-4.5	57.1	76.9	39.3	84.6	38.1	53.3	44.8	71.6
Gemini-3 Pro	60.7	81.2	53.6	86.2	57.1	75.1	57.1	80.8
Open-Source Foundation Models

Table 1: Main results across three real-world applications. We report Acc (binary task completion rate) and Score (average per-task score, 0–1). Best results per column are bolded . Green and red deltas indicate changes relative to Qwen3-32B.

Panel of Figure 3 — Figure 3: Effect of credit assignment. (a-b) Performance comparison of different credit assignment strategies on GitHub and Notion. (c-d) Distribution of mean credit and total credit for well-aligned (red) and misaligned (blue) trajectories on GitHub (c) and Notion (d).

Task Instruction	Acc.	Score
Original $I_{n}$	7.1	65.6
Inferred $\hat{I}_{n}$	14.2	70.1

Table 2: Effect of retrospective task inference on Notion.

Figure 4: Performance on Notion when progressively adding lower-quality data.

Method	Training Paradigm	GitHub		Notion		Postgres		Avg.
		Acc.	Score	Acc.	Score	Acc.	Score	Acc.	Score
Qwen3-14B	No additional training	7.1	48.2	0	48.8	19.0	31.3	8.7	42.8
Toucan-14B	SFT on 1.5M synthetic trajs.	0	45.2	0	42.5	0	11.4	0.0	33.3
AWM-14B	Online RL on synthetic envs.	7.1	45.4	0	44.5	23.8	38.0	10.3	42.6
AgentBrew-14B	Offline transfer	17.9	56.3	14.2	69.8	28.6	40.0	20.2	55.3

Table 3: Offline transferability on 14B models. AgentBrew distills data collected by Qwen3-32B into a 14B model via the offline pipeline, without additional environment interaction.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Env.	#Tools	Tools
GitHub	41	create_repository, get_repository, list_repositories, fork_repository, get_repository_languages, get_file_contents, create_or_update_file, delete_file, push_files, create_issue, get_issue, list_issues, update_issue, add_issue_comment, create_pull_request, get_pull_request, list_pull_requests, get_pull_request_files, create_pull_request_review, merge_pull_request, create_branch, list_branches, list_commits, list_workflow_runs, get_workflow_run, list_workflow_jobs, cancel_workflow_run, search_repositories, search_code, search_issues, search_users, list_code_scanning_alerts, get_code_scanning_alert, list_dependabot_alerts, get_dependabot_alert, list_secret_scanning_alerts, get_secret_scanning_alert, list_global_security_advisories, list_org_repository_security_advisories, list_repository_security_advisories, create_pull_request_with_copilot
Notion	19	get-user, get-users, get-self, post-database-query, post-search, get-block-children, patch-block-children, retrieve-a-block, update-a-block, delete-a-block, retrieve-a-page, patch-page, post-page, create-a-database, update-a-database, retrieve-a-database, retrieve-a-page-property, retrieve-a-comment, create-a-comment
Postgres	9	list_schemas, list_objects, get_object_details, execute_sql, explain_query, get_top_queries, analyze_workload_indexes, analyze_query_indexes, analyze_db_health

Appendix

Table 4: Tool sets exposed by each environment.

GitHub (14)		Notion (12)		PostgreSQL (15)
	Project bootstrap & self-review		Database creation & schema design		Bulk migration & transformation
	Multi-source code integration		Cross-database relation & sync		Hierarchy-aware CRUD & integrity
	Ecosystem metrics & reporting		Batch query, filter & migration		Dashboard reporting & aggregates
	Decision-based comparative forking		Batch updates & coordinated edits		SQL query debugging & fixing
	Simulated issue automation		Aggregate computation & dashboards		Query optimization & indexing
	Fork-and-fix contribution		Multi-source query & synthesis		Operational workflow management

Appendix

Table 5: Task categories generated for each environment.

Figure 5: Case study on the GitHub environment.

Figure 6: Case study on the Notion environment.

Figure 7: Case study on the PostgreSQL environment.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 290 papers

AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents
Jul 18, 2026Yangqin Jiang, Chao HuangLLM Agent Self-Improvement LLM Agent Skill Learning
The University of Hong Kong
Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use
Feb 23, 2026Ruocheng Guo, Kaiwen Dong, Xiang Gao +1LLM Tool Use LLM Agent Training
Intuit AI Research, Mountain View, CA, USA
Training Advisors for LLM Agents from Task Outcomes
Oct 7, 2026Sergei Polezhaev, Barys Liskavets, Ori Press +1LLM Agent Training RL for LLM Agents
Nebius AI

Jul 18, 2026J/K move · Enter open · S save

AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents

Yangqin Jiang, Chao Huang

The University of Hong Kong

LLM Agent Self-Improvement LLM Agent Skill Learning Knowledge Distillation Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Deploying LLM agents typically requires a compact test-time student, even if a stronger teacher is available during training. We study knowledge brewing: distilling a teacher's interactive experience into a persistent external memory for the student. Crucially, this requires no weight updates, expert demonstrations, ground-truth labels, or test-time teacher access. This setting poses two challenges: environments provide only sparse, binary feedback, and teacher-authored notes must be inherently tailored to be concretely executable by a substantially weaker student. To address these hurdles, we propose AgentBrew, comprising two coupled components. First, a failure-triggered teacher--Ralph Loop mitigates sparse feedback by transforming student failures into environment-validated notes. Second, student-aware synthesis calibrates teacher knowledge to the weak executor's operational granularity, yielding model-specific, actionable guidance. Extensive evaluations and comprehensive ablations across coding, math, and tool-use tasks demonstrate that this asymmetric, training-free brewing paradigm produces highly capable yet deployable LLM agents.

Abstract

Tool-Using Agents

Explore similar work

AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agents

Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use

Training Advisors for LLM Agents from Task Outcomes