AutoOnboarder: Autonomously Onboarding LLM Agents

Tanay Arora; Cong Liu

cs.AIOct 15, 2026

AutoOnboarder: Autonomously Onboarding LLM Agents

Authors: Tanay Arora, Cong Liu

Organizations: Research Science Institute, Massachusetts Institute of Technology · CliGenAILab, Boston Children’s Hospital

Abstract

Agentic large language models have been documented to lose reasoning capability when placed inside a harness, the software that mediates their observations and actions. To reconcile these losses, we introduce AutoOnboarder, an autonomous system that adapts an agent to a task the way a trainer onboards a new employee. It places the agent under the supervision of an intelligent ``Onboarder'' coding agent, which iteratively tailors its foundation model, prompt, and harness source code to simultaneously improve performance and decrease cost. Like a real onboarding process, the Onboarder is only given tens of examples, and all evaluation data is isolated on a separate virtual machine to prevent leakage. We report the lessons learned from building and testing AutoOnboarder on genomic-sequencing prior authorization form submission. Autonomously onboarded agents are able to match or exceed the performance of a Gemini 3.5 Flash baseline at up to 29 $\times$ lower costs. However, these agents remain imperfect, and the Onboarding process varies greatly in terms of final performance. We conclude that future Onboarding systems require a robust environment for the Onboarder, including a structured thought interface, (controlled) internet access, and a wide model search space.

Figures & tables

	May change
Family	Weights	Prompt	Model	Harness	Cost objective
Finetuning	✓	–	–	–	–
Prompt-adaptation	–	✓	–	–	–
Interface design	–	✓	–	✓	–
Agents improving agents	–	✓	–	✓	–
Onboarding	–	✓	✓	✓	✓

Table 1: Scopes of agent-adaptation families.

Figure 1: Heavily simplified visualization of the Executor’s evaluation task.

Figure 2: The two-machine architecture.

Cost goal ( $\beta$ )	Metric	Run 1	Run 2	Run 3	Std. Dev.	Range	Flips
Evaluation Noise
$\$ 0.10$	$S$	0.309	0.294	0.298	0.007	0.014	19
	MFA	0.423	0.487	0.455	0.032	0.064
	CH	0.617	0.589	0.596	0.015	0.028
	$/case	0.086	0.083	0.083	0.002	0.004
$\$ 0.04$	$S$	0.323	0.340	0.326	0.009	0.018	24

Table 2: Spread of validation metrics ( Section 2.2 ) across runs. Top: three validation repeats at varying cost goals. Bottom: three onboarding repeats at the

\beta=\

0.04

goal. Std. Dev. is the sample standard deviation calculated from the three repeats. Flips is the number of the 180 cases whose outcome (failure vs. success) is not constant across all three repeats. The Onboarding run at cost goal

$0.01$ was unable to produce a cheap-enough candidate.

Figure 3: Onboarding learning curves, displaying Per-Case Cost (left), MFA (middle), and CH (right) with respect to Onboarding iteration. Filled circles correspond to the nominated iteration, crosses correspond to iterations that were reverted and discarded, and open circles correspond to iterations that were kept, but not nominated. All reported calculations are performed on the Onboarding data, rather than the validation data. Iteration 3 of Run 3 was stopped by the Onboarder before scoring, so it is not plotted.

Figure 4: Decision flips of Onboarding runs by sample class type ( Appendix Appendix A ). Left (A): Flips in evaluation noise repeats, top bars correspond to

\beta=\

0.10

and bottom bars correspond to

\beta=$0.04$ . Right (B): Flips in Onboarding noise repeats.

Candidate	$S$	MFA	CH	Perfect subs. (/39)	Near-perfect subs. (/39)	$/case
Gemini 3.5 Flash (stock)	0.319	0.024	0.638	0	0	0.934
GPT-5.6 Luna (stock)	0.028	0.000	0.057	0	0	0.027
Gemini 3.1 Flash-Lite (stock)	0.000	0.393	0.000	0	0	0.041
GPT-4.1 Nano (stock)	0.000	0.000	0.000	0	0	0.042
PMH	0.323	0.406	0.645	0	2	0.069
PM	0.325	0.773	0.624	1	7	0.036

Table 3: Validation results ( Section 2.2 ) of the Onboarded candidates at

\beta=\

0.04$ and the stock baselines. Near-perfect submissions have exactly 36 of 37 fields correct.

Figure 5: Halting behavior by coverage group ( Section 3.4 ). For each stacked bar, the filled segment is the share halted for the correct reason, and the gray extension adds halts with an incorrect or technical reason, so the full bar is total halting. The separated bar in red reports halts on the 39 valid cases, which should have been submitted.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Group	Class: cases (% of dataset)	Definition
Valid	P : 27 (15.0%)	Valid, well-supported request.
39 (21.7%)	Dsec : 12 (6.7%)	Valid request with unrelated secondary details; should still submit.
Data-integrity (T)	T1a : 4 (2.2%); T1b : 5 (2.8%)	Subscriber listed as a parent is implausibly close in age to the patient, or born after the patient.
53 (29.4%)	T2 : 6 (3.3%)	Specimen collection date precedes the patient’s date of birth.
	T3a : 6 (3.3%); T3b : 6 (3.3%)	Two patients share an identical name (and, for T3b, date of birth), making the case ambiguous.
	T4a : 5 (2.8%); T4b : 5 (2.8%)	Provider or lab NPI, or provider phone or fax, is not valid.

Appendix

Table 4: Validation-set sample classes and prevalence. The set contains 39 valid cases (21.7%) and 141 deficient cases (78.3%); every subclass has at least two cases.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
Jun 30, 2026Zhaojian Yu, Penghao Yin, Shuzheng Gao +3Tool-Augmented Language Model Agents LLM Tool Use
1Tsinghua University · 2The Chinese University of Hong Kong · 3Simple Agent Lab
AgentOmnia: Scaling Agentic Models for Full-Scenario Applications
Jul 25, 2026Hao Jiang, Gangtao Xin, Yingdi Huang +35Tool-Augmented Language Model Agents Agentic RL
Huawei Cloud Post-Training Team
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
Sep 21, 2026Weihang Ding, Junfei ZhanLLM Agent Evaluation AI Agent Benchmarks
Department of Industrial Engineering and Operations Research University of California, Berkeley · Department of Computing Imperial College London, United Kingdom

Jun 30, 2026J/K move · Enter open · S save

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

Zhaojian Yu, Penghao Yin, Shuzheng Gao, Shilin He+2

1Tsinghua University · 2The Chinese University of Hong Kong · 3Simple Agent Lab

Tool-Augmented Language Model Agents LLM Tool Use LLM Agent Training LLM Post-Training Language Model Self-Improvement

Open paper AskPDF arXiv

Training language models (LMs) remains a highly human-intensive process, even as frontier language model agents become increasingly capable at software engineering and other long-horizon tasks. A central challenge is that autonomous post-training is not just a coding problem: it requires the agent to repeatedly plan iterations, construct benchmark-aligned data, run stable training jobs, evaluate checkpoints, and preserve experiment state across many hours of interaction. We present AutoTrainess, a LM agent that exposes these operations as a repository of agent-computer interfaces for planning, data preparation, training, evaluation, and logging. Rather than leaving the agent to operate in a raw CLI environment with an underspecified action space, AutoTrainess externalizes prior human experience as explicit workflows, rules, and execution constraints that guide the agent toward effective and reliable training behavior. On PostTrainBench, AutoTrainess consistently outperforms CLI-only baselines, achieving 26.94 average score with GPT-5.4 (Codex) versus 23.21 for CLI-only. It also generalizes across models and harnesses, improving DeepSeek-V4-Flash (OpenCode) from 12.13 to 19.58.

Abstract

LLM Agent Training

Explore similar work

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

AgentOmnia: Scaling Agentic Models for Full-Scenario Applications

Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers