Automatic Configuration of LLM Post-Training Pipelines

Channe Chwa; Xinle Wu; Yao Lu

cs.LGMar 19, 2026

Automatic Configuration of LLM Post-Training Pipelines

Authors: Channe Chwa, Xinle Wu, Yao Lu

Organizations: National University of Singapore

Abstract

LLM post-training pipelines that combine supervised fine-tuning and reinforcement learning are difficult to configure under realistic compute budgets: the configuration space is high-dimensional and heterogeneous, stages are strongly coupled, and each end-to-end evaluation is expensive. We propose AutoPipe, a budget-aware two-stage framework for configuration selection in LLM post-training. Offline, AutoPipe learns a dataset-conditioned learning-to-rank surrogate from historical runs, capturing within-dataset preferences and providing transferable guidance toward promising regions of the configuration space. Online, for a new dataset, AutoPipe uses the offline guidance to steer Bayesian optimization and models dataset-specific deviations with a Gaussian-process residual surrogate. To reduce evaluation cost, each trial is early-stopped and scored by a learned predictor that maps early training signals to a low-cost proxy for final post-training performance. Experiments on biomedical reasoning tasks show that AutoPipe consistently outperforms offline-only baselines and achieves comparable performance with the strongest online HPO baselines while using less than 10% of their computational cost.

Figures & tables

Figure 1: Overview of the proposed two-phase configuration selection framework.

Dataset	Reference
MQA	Ours
MQAR	Ours
PQA	Ours
MQA(R)	Ours
MQA(R)PQA	Ours
HuaTuo	( Chen et al., 2024 )

Table 1: Post-training datasets used in this work.

Benchmark	Reference
BioASQ (yes/no)	( Xiong et al., 2024 )
GPQA (Biology)	( Rein et al., 2024 )
MedBullets (4 options)	( Chen et al., 2025 )
MedBullets (5 options)	( Chen et al., 2025 )
MedMCQA	( Pal et al., 2022 )
MedQA	( Jin et al., 2021 )

Table 2: Downstream benchmarks in our evaluation.

Phase	Hyperparameter	Possible values
	Base model	Qwen2.5-1.5B-Instruct
		Qwen2.5-3B-Instruct
		Qwen2.5-7B-Instruct
		Llama3.2-1B-Instruct
		Llama3.2-3B-Instruct
		Llama3.1-8B-Instruct

Table 3: Configuration space searched by all methods.

Method	Performance	Regret	Cost
Static & Zero-Shot Baselines
Random (Single run)	0.391	0.237	3.62
LLM Rec. (Description-only)	0.307	0.321	7.60
LLM Rec. (History-conditioned)	0.442	0.186	5.59
Global	0.529	0.099	4.05
Offline-only (Ranker)	0.558	0.070	6.16

Table 4: LODO configuration selection performance under a fixed number of online evaluations using either full-fidelity signals or early-stop (ES) proxy signals. Performance is measured in average benchmark accuracy and cost in average H200 hours required.

Metric	Offline Ranker	ES Regressor
Regret	0.110	0.079
Recall@1	0.100	0.100
Recall@3	0.300	0.300
Recall@5	0.400	0.300
Recall@10	0.700	0.500
nDCG@5	0.853	0.862

Table 5: LODO ranking performance of offline surrogate and early-stop (ES) regressor.

Variant	Final Score	Regret	Cost
Full method	0.590	0.038	8.71
No Residual Formulation	0.516	0.112	7.88
No BO	0.527	0.101	9.14
No Offline	0.497	0.131	5.60

Table 6: Ablation results for main components.

Figure 2: Sensitivity of performance (average benchmark accuracy) to online evaluation budget.

Variant	Final Score	Regret	Cost
25%	0.565	0.063	7.02
50%	0.590	0.038	8.71
75%	0.583	0.045	9.58

Table 7: Sensitivity to early-stop truncation length.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Datasets
Method	1	2	3	4	5	6	7	8	9	10	Avg
Static & Zero-Shot Baselines
Random (Single run)	0.620	0.383	0.503	0.370	0.570	0.141	0.342	0.425	0.088	0.468	0.391
LLM Rec. (Description-only)	0.513	0.544	0.423	0.547	0.477	0.371	0.088	0.018	0.052	0.035	0.307
LLM Rec. (History-conditioned)	0.465	0.630	0.435	0.558	0.461	0.382	0.532	0.410	0.431	0.117	0.442
Global	0.556	0.628	0.632	0.624	0.642	0.622	0.454	0.659	0.018	0.451	0.529

Appendix

Table D.1: Configuration selection performance across ten held-out datasets under a fixed number of online evaluations. Performance is calculated as the average benchmark accuracy. Bold indicates the best performance per dataset; italics indicate the second-best.

	Datasets
Method	1	2	3	4	5	6	7	8	9	10	Avg
Static & Zero-Shot Baselines
Random (Single run)	0.003	0.247	0.141	0.313	0.084	0.506	0.264	0.234	0.417	0.157	0.237
LLM Rec. (Description-only)	0.110	0.086	0.221	0.136	0.177	0.276	0.518	0.641	0.453	0.590	0.321
LLM Rec. (History-conditioned)	0.158	0.000	0.209	0.125	0.193	0.265	0.074	0.249	0.074	0.508	0.186
Global	0.067	0.002	0.012	0.059	0.012	0.025	0.152	0.000	0.487	0.174	0.099

Appendix

Table D.2: Configuration selection regret across ten held-out datasets under a fixed number of online evaluations. Regret is calculated relative to the best-performing configuration in the reference pool for each dataset. Bold indicates the best performance (lowest regret) per dataset; italics indicate the second-best.

	Datasets
Method	1	2	3	4	5	6	7	8	9	10	Avg
Static & Zero-Shot Baselines
Random (Single run)	3.62	2.15	4.19	1.98	6.93	0.49	1.19	2.61	6.72	6.36	3.62
LLM Rec. (Description-only)	7.95	8.55	8.73	11.56	10.33	3.22	5.52	7.40	5.04	7.74	7.60
LLM Rec. (History-conditioned)	4.13	5.30	4.15	6.57	11.47	3.05	5.37	2.88	3.83	9.09	5.59
Global	3.58	6.14	3.77	4.67	4.52	3.77	3.62	2.76	3.48	4.19	4.05

Appendix

Table D.3: Configuration selection cost across ten held-out datasets under a fixed number of online evaluations. Cost is expressed in terms of H200 hours.

	Datasets
Metric	1	2	3	4	5	6	7	8	9	10	Avg
Offline Ranker
Regret	0.060	0.057	0.048	0.021	0.032	0.042	0.585	0.070	0.000	0.183	0.110
Recall@1	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	1.000	0.000	0.100
Recall@3	0.000	0.000	0.000	0.000	1.000	1.000	0.000	0.000	1.000	0.000	0.300
Recall@5	0.000	1.000	0.000	0.000	1.000	1.000	0.000	0.000	1.000	0.000	0.400

Appendix

Table D.4: Offline surrogate ranking performance across ten held-out datasets under LODO evaluation.

Figure D.1: Sensitivity of configuration selection performance to the online evaluation budget across ten held-out datasets.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive
May 12, 2026Taicheng Guo, Nitesh V. Chawla, Olaf Wiest +1Multi-Fidelity Optimization Large Language Model-Guided Optimization
University of Notre Dame
LLM Agents for Automated Post-Training via Structured Recipe Optimization
Oct 13, 2026Ruohong Liu, Bohan Tang, Lekang Jiang +2LLM Post-Training Large Language Model-Guided Optimization
LIGHTSPEED · University of Cambridge · Independent Researcher
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China

May 12, 2026J/K move · Enter open · S save

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive

Taicheng Guo, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang

University of Notre Dame

Multi-Fidelity Optimization Large Language Model-Guided Optimization LLM Agent Training Autonomous Experimentation

Open paper AskPDF arXiv

Effectively configuring scalable large language model (LLM) experiments, spanning architecture design, hyperparameter tuning, and beyond, is crucial for advancing LLM research, as poor configuration choices can waste substantial computational resources and prevent models from realizing their full potential. Prior automated methods are designed for low-cost settings where repeated trial and error is feasible, but scalable LLM experiments are too expensive for such extensive iteration. To our knowledge, no work has addressed the automation of high-cost LLM experiment configurations, leaving this problem labor-intensive and dependent on expert intuition. Motivated by this gap, we propose AutoLLMResearch, an agentic framework that mimics how human researchers learn generalizable principles from low-fidelity experiments and extrapolate to efficiently identify promising configurations in expensive LLM settings. The core challenge is how to enable an agent to learn, through interaction with a multi-fidelity experimental environment that captures the structure of the LLM configuration landscape. To achieve this, we propose a systematic framework with two key components: 1) LLMConfig-Gym, a multi-fidelity environment encompassing four critical LLM experiment tasks, supported by over one million GPU hours of verifiable experiment outcomes; 2) A structured training pipeline that formulates configuration research as a long-horizon Markov Decision Process and accordingly incentivizes cross-fidelity extrapolation reasoning. Extensive evaluation against diverse strong baselines on held-out experiments demonstrates the effectiveness, generalization, and interpretability of our framework, supporting its potential as a practical and general solution for scalable real-world LLM experiment automation.

Abstract

LLM Post-Training

Explore similar work

AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensive

LLM Agents for Automated Post-Training via Structured Recipe Optimization

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning