LLM Agents for Automated Post-Training via Structured Recipe Optimization

Ruohong Liu; Bohan Tang; Lekang Jiang; Jack Umenberger; Yiwen Guo

cs.AIOct 13, 2026

LLM Agents for Automated Post-Training via Structured Recipe Optimization

Authors: Ruohong Liu, Bohan Tang, Lekang Jiang, Jack Umenberger, Yiwen Guo

Organizations: LIGHTSPEED · University of Cambridge · Independent Researcher

Abstract

Automated post-training transforms pre-trained large language models (LLMs) into models capable of handling diverse domains by automatically constructing and optimizing a recipe of coupled decisions over data sources, data processing, cross-domain composition, and training configuration under a limited experimental budget. Existing automation either searches predefined recipe spaces that enable systematic optimization but remain fixed rather than adapting their available choices to the target tasks, or uses open-ended agents that discover task-specific choices but leave them implicit in trial-level edits, limiting their systematic reuse and recombination across trials. To fill this gap, we introduce optimization over a task-adapted recipe space, a novel paradigm that can adapt the recipe space itself to the target tasks while representing its choices explicitly for systematic reuse, recombination, and joint optimization. We instantiate this paradigm with Structured Recipe Optimization (SRO), comprising an LLM-based Recipe-Space Designer that constructs the task-adapted recipe space and a Multi-Domain Recipe Optimizer that uses structured trial evidence to guide LLM candidate construction and multi-objective Bayesian trial selection. Extensive experiments show that SRO achieves the highest average score in all evaluated target-model and agent-LLM combinations. In joint post-training across three capability domains and six benchmarks, SRO reaches 41.1 versus 15.4 for the strongest baseline on Qwen3-1.7B with GPT-5.4, and 53.8 versus 40.8 on Qwen3-4B with DeepSeek-V4-Flash. Our code is available at https://github.com/RuohLiuq/SRO.

Figures & tables

Figure 1: Three paradigms for automated post-training.

Figure 2: Overview of SRO. The Recipe-Space Designer constructs a task-adapted recipe space, which the Multi-Domain Recipe Optimizer searches by Domain-Structured Trial Memory and LLM-Augmented Multi-Objective Bayesian Optimization.

			Benchmark Scores ( $\uparrow$ )
Model	Agent LLM	Harness	GSM8K	GPQA Main	HumanEval	Avg. ( $\uparrow$ )
Qwen3-1.7B-Base	–	None	12.7	14.1	7.9	11.6
	DeepSeek-V4-Flash	PostTrainBench	49.0	27.5	54.9	43.8
		AutoTrainess	79.1	28.1	53.7	53.6
		SRO (Ours)	68.2	34.2	64.6	55.6
	GPT-5.4	PostTrainBench	48.4	28.6	39.0	38.7

Table 1: Single-domain post-training with two base models and two agent LLMs. Each run targets one benchmark; Avg. denotes the mean score across the three separate runs.

			Benchmark Scores ( $\uparrow$ )
			Mathematical Reasoning		Coding & Tool Use		Science & Health
Model	Agent LLM	Harness	AIME 2025	GSM8K	BFCL	HumanEval	GPQA Main	HealthBench	Avg. ( $\uparrow$ )
Qwen3-1.7B-Base	–	None	0.0	12.7	0.0	7.9	14.1	7.5	7.0
	DeepSeek-V4-Flash	PostTrainBench	0.0	58.8	92.0	35.4	27.9	8.6	37.1
		AutoTrainess	0.0	73.8	92.0	58.5	30.8	15.6	45.1
		SRO (Ours)	6.7	76.7	91.0	58.5	16.1	22.7	45.3

Table 2: Multi-domain post-training across six benchmarks in three domains. Each run jointly targets all six benchmarks. Avg. is the mean across domains.

		Benchmark Scores ( $\uparrow$ )
		Mathematical Reasoning		Coding & Tool Use		Science & Health
Model	Variant	AIME 2025	GSM8K	BFCL	HumanEval	GPQA Main	HealthBench	Avg. ( $\uparrow$ )
Qwen3-1.7B	Full SRO	6.7	76.7	91.0	58.5	16.1	22.7	45.3
	w/o structured recipe search	0.0	57.3	85.0	34.1	23.2	7.6	34.5
	w/o LLM proposer	6.7	74.3	94.0	55.5	20.3	13.4	44.0
	w/o EHVI selection	0.0	56.0	94.0	51.2	29.7	15.4	41.1

Table 3: Comparison with ablations of the recipe space design and search mechanisms. All runs use DeepSeek-V4-Flash and the same compute budget on each base model.

Figure 3: Search trajectory on HumanEval for Qwen3-1.7B-Base with DeepSeek-V4-Flash. Each point denotes an evaluated recipe, and the solid step curve tracks the best HumanEval pass@1 observed so far. The vertical dashed line separates the SFT and GRPO phases. Numbered annotations mark the seed SFT recipe, the introduction of Rewrite-generated data, the addition of Correction data, the first GRPO improvement, and the final selected checkpoint.

Figure 4: Optimization trajectories of SRO and its variants under the same compute budget. Each curve tracks the best multi-domain average obtained over completed trials.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Domain	Test size	Output contract	Scoring
AIME 2025	math	30	ANSWER: <integer>	exact match
GSM8K	math	1,319	ANSWER: <number>	exact match
BFCL v3	coding & tool use	500	structured tool_calls	schema execution
HumanEval	coding & tool use	164	complete Python function	unit tests
GPQA Main	science & health	448	ANSWER: <A--D>	exact match
HealthBench-Easy	science & health	245	free-form responses	LLM-judge rubrics

Appendix

Table 4: Evaluation benchmarks, output formats, and scoring rules for the single-domain and multi-domain experiments.

Model	AIME	GSM8K	BFCL	HumanEval	GPQA	HealthBench	Mean
Qwen3-1.7B-Base	0.0	12.7	0.0	7.9	14.1	7.5	7.0
Qwen3-1.7B-Instruct	26.7	88.5	94.0	68.9	35.5	44.9	59.7
Qwen3-4B-Base	3.3	41.8	0.0	36.6	13.4	13.4	18.1
Qwen3-4B-Instruct	53.3	93.8	95.0	77.4	44.6	52.7	69.5

Appendix

Table 5: Base and instruction-tuned reference scores used to normalize the BO objectives. Scores are reported on a percentage scale.

Hyperparameter	Value
Trees per domain	24
Maximum tree depth	4
Feature subsampling fraction	0.7
Monte-Carlo draws $S$	64
Warm-start threshold (feasible trials)	5
Warm-start weights (conf / novelty / risk)	0.4 / 0.2 / 0.3

Appendix

Table 6: Bayesian optimizer hyperparameters, held fixed wherever the corresponding components are retained.

Base model	Before training	Best SFT	GRPO	Final	SRO (Ours)
Qwen3-1.7B-Base	10.3	55.0	55.7	55.7	68.2
Qwen3-4B-Base	43.0	74.8	74.5	74.8	87.6

Appendix

Table 7: GSM8K accuracy (%) under optimization over a predefined recipe space with a fixed training dataset. Single-domain SRO results with DeepSeek-V4-Flash from Table 1 are reproduced for reference.

Benchmark	Base training sources	Rewrite source	Correction source
AIME 2025	AIME 1983–2024; Hendrycks MATH	AIME 1983–2024	aime_diag
GSM8K	GSM8K train; MetaMathQA (GSM subset)	GSM8K train	gsm8k_diag
HumanEval	MBPP; MBPP+ (EvalPlus)	MBPP	mbpp_diag
BFCL	Hermes function-calling; Glaive v2	Hermes	bfcl_hermes_diag
GPQA Main	MMLU (science subset); ARC	MMLU (science subset)	arc_diag
HealthBench	MedAlpaca flashcards; MedAlpaca WikiDoc	MedAlpaca flashcards	medical-o1-reasoning

Appendix

Table 8: Discovery-selected sources for DeepSeek-V4-Flash with Qwen3-1.7B. Correction uses separate source data that exclude evaluation examples.

Benchmark	Method	Description
AIME	Rewrite	Evolve AIME-adjacent competition problems into new integer-answer (0–999) problems with independent full reasoning.
AIME	Correction	Turn wrong AIME-style answers on independent source data into corrected full-reasoning integer-answer solutions.
GSM8K	Rewrite	Evolve GSM8K-style word problems into new multi-step arithmetic problems with independent solutions.
GSM8K	Correction	Turn wrong GSM8K-style answers on independent source data into corrected full-reasoning numeric solutions.
HumanEval	Rewrite	Evolve MBPP-style tasks into new function-writing specs with independent solutions and executable tests.
HumanEval	Correction	Repair failing MBPP-style solutions on independent source data into correct, test-passing implementations.

Appendix

Table 9: Agent-designed data-processing methods in the DeepSeek-V4-Flash run for Qwen3-1.7B. Rewrite uses separate task-generation and answer-generation prompts; Correction generates supervision from model errors on independent source data.

Figure 5: Best-so-far search-time six-benchmark mean versus completed trial index for SRO and three optimizer ablations. Left: Qwen3-1.7B-Base. Right: Qwen3-4B-Base. All runs use DeepSeek-V4-Flash and the same compute budget. Trial counts differ because recipe execution costs vary.

Agent LLM	Base model	Trials	Selected trial	Search-time	Final
DeepSeek-V4-Flash	Qwen3-1.7B	16	11	45.3	45.3
DeepSeek-V4-Flash	Qwen3-4B	9	8	54.0	53.8
GPT-5.4	Qwen3-1.7B	17	7	40.9	41.1
GPT-5.4	Qwen3-4B	8	8	48.3	47.5

Appendix

Table 10: Search summary of the four SRO multi-domain runs. The selected trial maximizes the search-time six-benchmark mean. Final scores are obtained by evaluating the selected checkpoint on the complete benchmark sets after the time budget. Evaluation settings are detailed in Appendix B.1 . All scores are reported on a 0–100 scale.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration
Apr 15, 2026Zerun Ma, Guoqiang Wang, Xinchen Xie +7LLM Fine-Tuning LLM Agent Training
Shanghai AI Laboratory · Fudan University
PostTrainBench: Can LLM Agents Automate LLM Post-Training?
Mar 9, 2026Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4LLM Post-Training Autonomous Research Agents
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2
AutoAdapt: An Automated Domain Adaptation Framework for LLMs
Mar 9, 2026Sidharth Sinha, Anson Bastos, Xuchao Zhang +3Domain Adaptation for LLMs Domain Adaptation
Microsoft, M365 Research, India · Microsoft, M365 Research, Redmond · Microsoft Research, India

Apr 15, 2026J/K move · Enter open · S save

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

Zerun Ma, Guoqiang Wang, Xinchen Xie, Yicheng Chen+6

Shanghai AI Laboratory · Fudan University

LLM Fine-Tuning LLM Agent Training Multi-Agent LLM Systems

Open paper AskPDF arXiv

While Large Language Models (LLMs) have empowered AI research agents to perform isolated scientific tasks, automating complex, real-world workflows, such as LLM training, remains a significant challenge. In this paper, we introduce TREX, a multi-agent system that automates the entire LLM training life-cycle. By orchestrating collaboration between two core modules-the Researcher and the Executor-the system seamlessly performs requirement analysis, open-domain literature and data research, formulation of training strategies, preparation of data recipes, and model training and evaluation. The multi-round experimental process is modeled as a search tree, enabling the system to efficiently plan exploration paths, reuse historical results, and distill high-level insights from iterative trials. To evaluate the capability of automated LLM training, we construct FT-Bench, a benchmark comprising 10 tasks derived from real-world scenarios, ranging from optimizing fundamental model capabilities to enhancing performance on domain-specific tasks. Experimental results demonstrate that the TREX agent consistently optimizes model performance on target tasks.

Abstract

LLM Post-Training

Explore similar work

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

PostTrainBench: Can LLM Agents Automate LLM Post-Training?

AutoAdapt: An Automated Domain Adaptation Framework for LLMs