While Large Language Models (LLMs) have empowered AI research agents to perform isolated scientific tasks, automating complex, real-world workflows, such as LLM training, remains a significant challenge. In this paper, we introduce TREX, a multi-agent system that automates the entire LLM training life-cycle. By orchestrating collaboration between two core modules-the Researcher and the Executor-the system seamlessly performs requirement analysis, open-domain literature and data research, formulation of training strategies, preparation of data recipes, and model training and evaluation. The multi-round experimental process is modeled as a search tree, enabling the system to efficiently plan exploration paths, reuse historical results, and distill high-level insights from iterative trials. To evaluate the capability of automated LLM training, we construct FT-Bench, a benchmark comprising 10 tasks derived from real-world scenarios, ranging from optimizing fundamental model capabilities to enhancing performance on domain-specific tasks. Experimental results demonstrate that the TREX agent consistently optimizes model performance on target tasks.
Figures & tables
Figure 1: Illustration of TREX research progress on molecule generation. Top Left: The system takes the task definition as input, including task descriptions, benchmarks, and other information. Top Right: The model performance improves as TREX’s automated research progresses. Bottom left: Details of the key steps. Bottom right: Comparison of TREX (based on Qwen3-1.7B) with human-expert fine-tuned models based on Llama-series baselines.
Figure 2: The TREX framework. The system performs task-driven model optimization through iterative experimentation. The experimental trajectory is modeled as a tree, with node expansion guided by MCTS policy. Each iteration is driven by dual-agent collaboration: The Researcher designs the experimental plan and delegates the specific steps to the Executor . The Executor performs data processing, model training, and evaluation on a GPU cluster and returns empirical results to the Researcher for diagnostics. To support this workflow, the agents are equipped with a tailored toolset to access external knowledge and resources, and leverage condensed memory contexts to preserve the continuity and efficiency of the iterative process.
Molecule generation : edit, optimize, and generate molecules following textual instructions
oMeBench [ 49 ]
Chemical Mechanism Reasoning : predict stepwise intermediates and electron-transfer steps for organic reactions
HoC [ 1 ]
Cancer Hallmark Classification : assign cancer hallmark labels to biomedical publication abstracts
CS-Bench [ 39 ]
Computer Science Proficiency Evaluation : answer knowledge and reasoning questions across 26 subfields of computer science
OpenFinData [ 8 ]
Financial Question Answering : solve real-world financial tasks including numerical calculation, knowledge retrieval, and chart understanding
Table 1: LLM fine-tuning tasks in FT-Bench. Detailed task definitions are provided in Appendix A.1
Benchmark
#Tasks (LLM-FT / Total)
Evaluation Focus
MLAgentBench [ 14 ]
0/13
End-to-end machine learning experimentation with iterative model development and code refinement.
ScienceAgentBench [ 4 ]
0/102
Fine-grained scientific workflow tasks derived from real research across multiple disciplines.
RE-Bench [ 47 ]
1/7
Open-ended machine learning research engineering tasks with direct comparison to human experts.
MLEBench [ 2 ]
2/75
Real-world machine learning engineering tasks from Kaggle competitions with human leaderboard baselines.
MLGymBench [ 32 ]
0/13
AI research tasks spanning multiple domains, with a Gym-style interface and supporting RL-based agent training.
FT-Bench (ours)
10/10
End-to-end LLM fine-tuning tasks derived from real-world research and application scenarios.
Table 2: Comparison of benchmarks for autonomous research systems.
Task
Metric †
Ref Model
Base Model
TREX
Qwen3-235B-2507
Qwen3-1.7B
w/ Qwen3-Next-80B-Thinking
w/ Gemini 3 Pro
ACI-Bench
Rouge-1
0.240
0.205
0.260 (+157%)
0.502 (+849%)
TOMG-Bench
Val. & Acc.
0.642
0.182
0.557 (+ 82%)
0.681 (+108%)
oMeBench
oMeScore
0.283
0.198
0.392 (+228%)
0.484 (+336%)
HoC
Macro-F1
0.645
0.462
0.896 (+237%)
0.897 (+238%)
CS-Bench
Acc.
0.853
0.532
0.572 (+ 12%)
0.581 (+ 15%)
Table 3: Performance of the TREX framework for autonomous LLM fine-tuning on FT-Bench. We employ Qwen3-1.7B as the base model across all 10 tasks and compare Qwen3-Next-80B-Thinking and Gemini 3 Pro as the Researcher backend. To mitigate variance in task difficulty and metric scales, relative gains (in parentheses) are normalized to the performance gap between a strong reference (Qwen3-235B-Thinking) and the base model.
Action Strategy
w/ Qwen3-Next-80B-Thinking
w/ Gemini 3 Pro
A
S
I
A
S
I
Establish Baseline
10
10
–
10
10
–
Refine Data Pipeline
117
86
9
91
76
16
Construct Synthetic Data
33
22
2
50
45
9
Adjust Training Scheme
50
39
6
59
52
13
Table 4: Frequency distribution of strategies proposed by TREX during the FT-Bench evaluation. (A: Attempts, S: Successes, I: Improvements)
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Operator
Function Description
Loader
load_local_dataset
Imports datasets from local storage.
load_remote_dataset
Retrieves datasets from remote repositories (e.g., Hugging Face).
Scorer
compute_perplexity
Calculates text perplexity.
score_dataset_with_llm
Employs an LLM as a judge to evaluate data.
Generator
generate_text_embeddings
Computes dense vector representations for dataset entries.
generate_dataset_with_llm
Synthesizes Instruction-Response pairs from seed data.
Appendix
Table 5: Overview of Primitive Operators in AIDP. The toolkit provides a comprehensive set of operators for data ingestion, formation, analysis, synthesis, and selection.
Figure 7: Comparative performance trajectories of iterative experimental refinement using distinct LLM engines for tasks oMeBench, ACI-Bench, GTA, and HOC. The left panel illustrates the score progression with Gemini 3 pro as the Researcher, while the right panel depicts the corresponding progression with Qwen3-next-80b-thinking.
Figure 8: Comparative performance trajectories of iterative experimental refinement using distinct LLM engines for the task LawBench and EconlogicQA, SST-2 and CS-Bench. The left panel illustrates the score progression with Gemini 3 pro as the Researcher, while the right panel depicts the corresponding progression with Qwen3-next-80b-thinking.