FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents

Qizheng Li; Yifei Zhang; Xiao Yang; Xu Yang; Zhuo Wang; Weiqing Liu; Jiang Bian

cs.AIMar 2, 2026

FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents

Authors: Qizheng Li, Yifei Zhang, Xiao Yang, Xu Yang, Zhuo Wang, Weiqing Liu, Jiang Bian

Organizations: Microsoft Research Asia · Peking University · Nanjing University · The University of Chicago

Abstract

Fine-tuning large language models for vertical domains remains labor-intensive, requiring practitioners to curate data, configure training, and iteratively diagnose model behavior. Despite growing interest in autonomous machine learning and language agents, end-to-end LLM fine-tuning has not been systematically studied as an interactive agent task. We introduce FT-Dojo, an interactive benchmark environment for autonomous LLM fine-tuning, comprising 13 tasks across 5 domains. Rather than a new collection of static datasets, FT-Dojo standardizes a task interface, shared raw-data repository, sandboxed execution environment, structured feedback protocol, and held-out evaluation procedure. We further develop FT-Agent, a fine-tuning-oriented autonomous framework that uses structured iteration planning, fail-fast validation, and multi-level feedback analysis to refine data and training strategies. Experiments show that FT-Agent provides a strong initial baseline, achieving the best performance on 10 out of 13 tasks, with additional controlled comparisons against frontier agents, open-source planning backbones, and multi-run statistics supporting the main findings. Case studies show that agents can recover from failures through cumulative learning, while still exposing limitations in causal diagnosis and long-horizon planning. The implementation is available at https://github.com/microsoft/rd-agent.

Figures & tables

Figure 1: Overview of FT-Dojo architecture and agent interaction workflow. Agents operate within an isolated sandbox and interact with three core interfaces: Meta API $\mathcal{I}$ for task and system information, Data Repository $\mathcal{D}$ for training data, and Evaluator $\mathcal{F}$ that returns structured feedback. Agents iteratively query information, submit code and configurations, execute training, and refine based on evaluation feedback until achieving satisfactory performance.

Domain	Task	Description	Train Data Quality	Metrics	Train	Valid/Test
Mathematics	AIME 2025	Competition-level mathematical reasoning requiring multi-step symbolic derivations, algebraic manipulation, and exact numerical answers	82% lack solution No CoT	Accuracy	40315	15
Patent Examination	Prior Art Retrieval	Given a patent claim and candidate patents, identify which were cited as prior art for rejecting the claim	No CoT Q&A pairs only High complexity	Acc, Macro-F1	54028	100
	Novelty Classification	Classify patent claims as allowed, §102 rejection (lacks novelty), or §103 rejection (obvious)		Acc, Macro-F1	136211	100
	Paragraph Identification	Identify the most relevant prior art paragraph from candidates that supports claim rejection		Accuracy	64210	100
Chemistry	Molecular Understanding	Predict molecular properties from SMILES: functional group counting, scaffold extraction, equivalence	Structured CoT Expert validated	MAE, Tanimoto	6319	160
	Molecule Editing	Modify molecular structures by adding, deleting, or substituting functional groups		Acc, Tanimoto	4497	50

Table 1: Task suite in FT-Dojo spanning 13 tasks across 5 domains. Data Quality column describes training data characteristics relevant for fine-tuning. Valid/Test indicates the size of both validation and test sets (identical for each task). See Appendix A for metric definitions and abbreviations.

Figure 2: Overview of FT-Agent. The agent iteratively explores configurations, learning from failed trials (red) to reach successful ones (green). Each iteration follows three stages: Strategy Proposal, Implementation & Validation, and Feedback Analysis.

	Math	Patent Examination			Table QA				Finance	Chemistry
	AIME	PAR4PC	NOC4PC	PI4PC	DA	FC	NR	Vis.	QA	Mol_Und			Mol_Edit	Mol_Opt		Reaction
Method	Acc $\uparrow$	F1 $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	P@1 $\uparrow$	Acc $\uparrow$	MAE $\downarrow$	TMS $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	SR $\uparrow$	VS $\uparrow$	FTS $\uparrow$	Acc $\uparrow$
Qwen2.5-7B-Instruct	0.00	57.78	26.20	34.00	32.78	75.00	32.00	4.00	65.10	0.53	0.09	63.67	22.20	23.33	68.67	15.59	30.00
Manual SFT	0.00	67.61	33.39	28.00	28.80	70.83	43.00	8.00	71.68	0.65	0.00	36.67	0.00	9.67	31.00	0.00	10.00
OpenHands	0.00	50.51	28.44	44.00	23.22	66.67	40.00	24.00	64.73	0.65	0.10	68.00	40.00	31.00	80.67	35.59	60.00
FT-Agent	11.11 $\pm$ 3.82	67.61 $\pm$ 2.40	36.81 $\pm$ 4.13	49.00 $\pm$ 4.36	34.71 $\pm$ 0.28	81.25 $\pm$ 2.08	45.33 $\pm$ 1.15	29.33 $\pm$ 6.11	67.20 $\pm$ 0.31	0.23 $\pm$ 0.16	0.36 $\pm$ 0.06	67.55 $\pm$ 5.68	54.44 $\pm$ 2.94	34.44 $\pm$ 2.17	90.00 $\pm$ 4.41	30.92 $\pm$ 11.22	35.33 $\pm$ 3.06

Table 2: Main results comparison. We compare FT-Agent against baselines across 13 tasks in 5 domains. FT-Agent achieves the best performance on 10 out of 13 tasks. Bold indicates best performance. FT-Agent results are reported as mean

\pm

std over 3 independent runs. Manual SFT is a constrained, resource-matched manual baseline without LLM assistance. See Appendix A for metric definitions and abbreviations.

Metric	OpenHands	FT-Agent
Avg. Loops	3.69	8.77
Avg. Improve Rate	17.31%	24.37%
Avg. Cost (iterative only)	$3.92	$5.73

Table 3: Exploration dynamics between OpenHands and FT-Agent.

Figure 3: Ablation study results. (a) Data scaling : Performance comparison between 2k and 5k training samples. (b) Planning backbone : FT-Agent variants with proprietary/open-source planners. (c) Target model scale : Performance across target sizes. Panel (b) shows FT-Agent variants only; cross-agent baselines are in Table 4 , with full ablations in Appendix Table 10 .

Method	Setting	AIME	PI4PC	Mol_Edit	Finance	Vis.	Avg.
Base Model	–	0.00	34.00	22.20	65.10	4.00	25.06
Manual SFT	LLM synthesis	20.00	28.00	50.00	67.57	20.00	37.11
OpenHands	GPT-5.2	0.00	44.00	40.00	64.73	24.00	34.55
Codex	GPT-5.2	6.67	42.00	40.00	65.63	26.00	36.06
Claude Code	Sonnet-4.6-thinking	0.00	51.00	44.00	68.32	36.00	39.86
FT-Agent	Qwen3.5-397B-A17B	6.67	24.00	34.00	67.16	28.00	31.97

Table 4: Five-task controlled comparison. Frontier agents, FT-Agent planning backbones, and an assisted-human reference on representative tasks. Scores are percentages; Avg. is over displayed tasks.

Panel of Figure 4 — Figure 4: Contrasting Learning Trajectories. (a) On Chemistry, the agent demonstrates cumulative learning , recovering from failure and progressively refining its approach through domain tool integration and iterative optimization. (b) On Patent Classification, it exhibits shotgun debugging , cycling through advanced techniques without identifying the root cause of overfitting.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

1. ChemCoTBench (Chemistry)
Task Category	Subtask	Total	Valid	Test
Mol_Und	Murcko_scaffold	40	20	20
	equivalence	100	50	50
	fg_count	100	50	50
	ring_count	20	10	10
	ring_system_scaffold	60	30	30

Appendix

Table 5: Detailed statistics of evaluation datasets across FT-Dojo benchmarks.

Method	AIME	PI4PC	Table Vis.	Finance QA	Mol_Edit
Manual SFT (original)	0.00	28.00	8.00	71.68	0.00
Manual SFT (w/ LLM synthesis)	20.00	28.00	20.00	67.57	50.00
FT-Agent	13.33	44.00	36.00	67.53	53.33

Appendix

Table 6: Manual SFT with LLM-assisted synthesis on five representative tasks. All scores are percentages and higher is better. FT-Agent values are representative single-run scores for this focused comparison.

Domain	Task	Metric	OpenHands		FT-Agent
Domain	Task	Metric	Val	Test	Val	Test
Math	AIME	Acc	0.00	0.00	20.00	13.30
Patent Exam.	PAR4PC	Exact Match	33.00	34.00	54.00	50.00
	PAR4PC	Macro F1	50.93	50.51	75.43	70.34
	NOC4PC	Acc	35.35	34.34	40.00	44.00
	NOC4PC	Macro F1	27.47	28.44	37.18	41.09

Appendix

Table 7: Full performance comparison on Validation and Test sets. This table supplements the main text by including all auxiliary metrics (e.g., Macro F1, Tanimoto, BLEU) and validation splits to demonstrate robustness. Bold indicates the best performance in each column (for error metrics like MAE, lower is better).

Task	OpenHands	FT-Agent
AIME	2	5
PAR4PC	3	6
Table QA FC	2	7
Mol_Opt	4	6
Average	2.75	6.0

Appendix

Table 8: Number of effective iterations (completing full training) within the 12-hour budget. FT-Agent achieves 2.2

\times

more effective iterations due to fail-fast validation.

Figure 5: Cross-domain data strategies autonomously discovered by FT-Agent. Left: for Numerical Reasoning, the agent mixes TableInstruct (90%) with out-of-domain DeepScaleR (10%). Right: for Mol_Opt, the agent combines three chemistry sub-tasks with task-informed proportions.

Figure 6: Visualization: training samples and Pass@1 over loops. Loop 8’s multi-stage filtering pipeline reduced training data from

\sim

1,800 to just 27 samples, causing Pass@1 to collapse from 20% to 4%.

Loop	Acc.	Strategy (Before)	Feedback (After)
0	34%	Gold-anchored rationale + LoRA	“accuracy improves from 29% baseline to 34%…eval loss improving early then degrading suggests overfitting”
1	23%	+NEFTune + longer examiner rationales (“to reduce overfitting”)	“NEFTune noise likely pushed the model to learn fluent justifications rather than improve discriminative paragraph choice”
2	15%	Remove NEFTune, keep contrastive rationales	“train–eval objective mismatch where model learns plausible narratives without reliably performing discriminative selection”
3	8%	Strict quote-matching filter (“improve data quality”)	“only 1111/2000 samples survived…set appears extremely easy…which can train toward overlap shortcuts”
4	30%	Hard-case sampling by lexical overlap	“shortfall consistent with supervision/distribution mismatch…sampled difficulty still concentrates in very-easy cases”
5	32%	Quota-based difficulty sampler	“hard-case quota did not materialize (bucket ‘ $\geq$ 7’ ended up 0)…shift likely didn’t target hardest regimes”

Appendix

Table 9: PI4PC: Complete decision trajectory over 8 loops. Each row shows the agent’s strategy (before execution) and feedback (after execution). Despite diverse interventions, the agent returns to baseline performance. Note the hedging language in feedback (“likely,” “appears”) indicating correlation-based rather than causal reasoning.

Domain	Task (Metric)	7B Model Settings				3B Model Settings
Domain	Task (Metric)	Base	Default (2k)	Scaling-5k	w/ GPT-4o	Base	FT
Math	AIME25 (Acc)	0.00/0.00	20.00 / 13.33	6.67/ 13.33	6.67/ 13.33	0.00/0.00	6.67/0.00
Patent	PI4PC (Acc)	29.00/34.00	40.00 /44.00	36.00/ 48.00	38.00/29.00	19.00/21.00	31.00/34.00
Finance	QA (Acc)	73.00/65.10	75.00/67.53	75.55 /66.33	70.33/ 70.95	61.44/56.98	60.73/58.47
Chem	Mol_Edit (Acc)	34.40/22.20	62.22/53.33	63.33 / 58.89	62.22/50.00	13.44/14.44	53.93/54.55
Table	Vis (Pass@1)	8.00/4.00	20.00/ 36.00	20.00/28.00	24.00 /20.00	0.00/8.00	8.00/24.00

Appendix

Table 10: Ablation study and baseline comparison organized by model size (7B vs. 3B). Scores are reported as validation/test . The Default setting represents FT-Agent with Qwen2.5-7B (2k samples, GPT-5.2 backbone). Scaling-5k and w/ GPT-4o are ablations on top of the 7B Default. Bold indicates the best validation and best test performance per row independently.

	Math	Patent Examination			Table QA				Finance	Chemistry
	AIME	PAR4PC	NOC4PC	PI4PC	DA	FC	NR	Vis.	QA	Mol_Und			Mol_Edit	Mol_Opt		Reaction
Method	Acc $\uparrow$	F1 $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	P@1 $\uparrow$	Acc $\uparrow$	MAE $\downarrow$	TMS $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	SR $\uparrow$	VS $\uparrow$	FTS $\uparrow$	Acc $\uparrow$
Qwen2.5-7B-Instruct	0	60.63	27.40	29.00	28.87	83.30	40.00	8.00	73.00	0.52	0.09	56.00	34.40	21.00	66.00	14.93	36.00
Manual SFT	0	68.00	27.11	22.00	34.14	72.92	45.00	20.00	73.14	0.44	0.00	46.00	2.23	12.67	31.67	0	10.00
OpenHands	0	50.93	27.47	47.00	33.78	79.17	45.00	8.00	74.40	0.54	0.08	59.00	55.56	34.67	81.67	29.97	64.00
FT-Agent	13.32 $\pm$ 6.67	67.50 $\pm$ 6.89	34.44 $\pm$ 4.82	38.67 $\pm$ 2.31	33.56 $\pm$ 3.16	82.64 $\pm$ 4.82	46.67 $\pm$ 4.62	16.00 $\pm$ 4.00	74.04 $\pm$ 1.10	0.24 $\pm$ 0.17	0.26 $\pm$ 0.08	62.78 $\pm$ 5.96	57.04 $\pm$ 5.01	36.22 $\pm$ 2.36	88.78 $\pm$ 6.38	30.64 $\pm$ 11.93	39.33 $\pm$ 3.06

Appendix

Table 11: Complete validation set results. We report validation performance across 13 tasks in 5 domains. FT-Agent results are reported as mean

\pm

std over 3 independent runs. Bold indicates the best performance.

\uparrow

: higher is better;

\downarrow

: lower is better. Abbreviations: DA/FC/NR = Data Analysis/Fact Checking/Num. Reasoning; P@1 = Pass@1; MAE = Mean Absolute Error (for FG/Ring Count); TMS = Tanimoto Similarity; SR = Success Rate; VS = Valid SMILES; FTS = Fingerprint Similarity.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration
Apr 15, 2026Zerun Ma, Guoqiang Wang, Xinchen Xie +7LLM Fine-Tuning LLM Agent Training
Shanghai AI Laboratory · Fudan University
AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
Jun 30, 2026Zhaojian Yu, Penghao Yin, Shuzheng Gao +3Tool-Augmented Language Model Agents LLM Tool Use
1Tsinghua University · 2The Chinese University of Hong Kong · 3Simple Agent Lab
Can RL Improve Generalization of LLM Agents? An Empirical Study
Mar 12, 2026Zhiheng Xi, Xin Guo, Jiaqi Liu +11LLM Agents RL for LLM Agents
Fudan University · Shanghai Artificial Intelligence Laboratory · Meituan

Apr 15, 2026J/K move · Enter open · S save

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

Zerun Ma, Guoqiang Wang, Xinchen Xie, Yicheng Chen+6

Shanghai AI Laboratory · Fudan University

LLM Fine-Tuning LLM Agent Training Multi-Agent LLM Systems

Open paper AskPDF arXiv

While Large Language Models (LLMs) have empowered AI research agents to perform isolated scientific tasks, automating complex, real-world workflows, such as LLM training, remains a significant challenge. In this paper, we introduce TREX, a multi-agent system that automates the entire LLM training life-cycle. By orchestrating collaboration between two core modules-the Researcher and the Executor-the system seamlessly performs requirement analysis, open-domain literature and data research, formulation of training strategies, preparation of data recipes, and model training and evaluation. The multi-round experimental process is modeled as a search tree, enabling the system to efficiently plan exploration paths, reuse historical results, and distill high-level insights from iterative trials. To evaluate the capability of automated LLM training, we construct FT-Bench, a benchmark comprising 10 tasks derived from real-world scenarios, ranging from optimizing fundamental model capabilities to enhancing performance on domain-specific tasks. Experimental results demonstrate that the TREX agent consistently optimizes model performance on target tasks.

Abstract

LLM Fine-Tuning

Explore similar work

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

Can RL Improve Generalization of LLM Agents? An Empirical Study