AutoAdapt: An Automated Domain Adaptation Framework for LLMs

Sidharth Sinha; Anson Bastos; Xuchao Zhang; Akshay Nambi; Chetan Bansal; Saravan Rajmohan

cs.LGMar 9, 2026

AutoAdapt: An Automated Domain Adaptation Framework for LLMs

Authors: Sidharth Sinha, Anson Bastos, Xuchao Zhang, Akshay Nambi, Chetan Bansal, Saravan Rajmohan

Organizations: Microsoft, M365 Research, India · Microsoft, M365 Research, Redmond · Microsoft Research, India

Abstract

Large language models (LLMs) excel in open domains but struggle in specialized settings with limited data and evolving knowledge. Existing domain adaptation practices rely heavily on manual trial-and-error processes, incur significant hyperparameter complexity, and are highly sensitive to data and user preferences, all under the high cost of LLM training. Moreover, the interactions and transferability of hyperparameter choices across models/domains remain poorly understood, making adaptation gains uncertain even with substantial effort. To solve these challenges, we present AutoAdapt, a novel end-to-end automated framework for efficient and reliable LLM domain adaptation. AutoAdapt leverages curated knowledge bases from literature and open-source resources to reduce expert intervention. To narrow the search space, we design a novel multi-agent debating system in which proposal and critic agents iteratively interact to align user intent and incorporate data signals and best practices into the planning process. To optimize hyperparameters under tight budgets, we propose AutoRefine, a novel LLM-based surrogate that replaces costly black-box search. Across 10 tasks, AutoAdapt achieves a 25% average relative accuracy improvement over state-of-the-art Automated Machine Learning baselines with minimal overhead.

Figures & tables

Figure 1: AutoAdapt takes as input the user instruction and dataset and delivers the domain adapted LLM based on user constraint

Framework	Key Functionality
Framework	Planning	Verification	Efficient Optimization	Effective LLM Adaptation	Training-Free Search	With Retrieval
AutoML-GPT ( Zhang et al., 2023 )	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
Prompt2Model ( Viswanathan et al., 2023 )	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$
HuggingGPT ( Shen et al., 2023 )	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
CAAFE ( Hollmann et al., 2023 )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
MLCopilot ( Zhang et al., 2024 )	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$

Table 1: Comparison between AutoAdapt and existing LLM-based frameworks.

Figure 2: Framework Overview: AutoAdapt processes user data and task definitions, integrates best practices, generates an executable training pipeline, and refines an intermediate model to produce a deliverable model for users.

Figure 3: Hierarchical Configuration Determination Using a Domain Adaptation Configuration Graph (ACG)

Figure 4: Success Rate (SR), Normalized Performance Score (NPS), Cumulative Score (CS) comparing AutoAdapt with baseline methods across datasets. Higher score indicates better results. AutoAdapt outperforms SoTA baselines across datasets. Detailed results are in $\lx@sectionsign$ I .

Dataset	AutoAdapt	AutoMLAgent	MLCopilot	HF Defaults	DS-Agent
UC Berkeley MATH	$27.8\pm 3.5$	$26.8\pm 2.4$	$27.2\pm 2.3$	$26.2\pm 0.06$	$\mathbf{30.0\pm 3.7}$
MedQA	$\mathbf{61.27\pm 1.1}$	$58.84\pm 1.3$	$57.50\pm 1.2$	$58.13\pm 0.04$	$59.86\pm 1.0$
CaseHold	$\mathbf{89.22\pm 0.9}$	$84.72\pm 1.1$	$82.63\pm 1.0$	$86.21\pm 0.08$	$87.28\pm 0.9$
PW	$\mathbf{67.69\pm 1.3}$	$64.53\pm 1.4$	$63.71\pm 1.5$	$65.78\pm 0.02$	$63.40\pm 1.6$
RCA	$\mathbf{2.30\pm 0.76}$	$2.23\pm 0.81$	$2.18\pm 0.77$	$2.23\pm 0.81$	$2.25\pm 0.66$
ARC	$\mathbf{83.79\pm 1.2}$	$77.39\pm 1.1$	$77.56\pm 1.0$	$78.24\pm 0.03$	$77.47\pm 1.4$

Table 2: Performance comparison across datasets and methods in the template aware (TA) setting. AutoAdapt is better than the baselines even when model and technique (SFT) is fixed.

Panel of Figure 5 — Figure 5: (a) Ablation and (b) hyperparameter study varying number of multi-agent debate rounds and AutoRefine trials.

Panel of Figure 6 — Figure 6: (a) RAG, DPO Results (b) Accuracy (bars) and wall-clock time (lines) by coreset sizes for HPO and AutoRefine.

Panel of Figure 7 — Figure 7: Top figure shows cost breakdown for MedQA across different agents in the MultiAgent planning phase. Bottom figures show the minimal time/money overhead and performance gains.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Adaptation Configuration Graph (ACG) for the LLM adaptation pipeline with example parameter values at each parameter node.

Figure 9: Best practices retrieval pipeline from KB.

Panel of Figure 10 — Figure 10: (Left) Boxplot showing the dispersion index metric for each dataset. We observe a median score around $0.5$ , indicating a balance between consistency and diversity in the recommendations. (Right) Boxplot showing relevance scores of the retrieved best practices. We observe consistently high relevance across datasets.

Panel of Figure 11 — Figure 11: Example hyperparameter range recommended by the multi-agent pipeline. Numeric parameters are expressed as intervals, while categorical parameters include discrete choices.

Dataset	Method of Evaluation	Metric
UC Berkeley MATH	OpenAI/simple_evals	Accuracy
MedQA	LLM-based equivalence matching (P1)	Accuracy
CaseHold	LLM-based equivalence matching (P1)	Accuracy
PEM	LLM-based answer extraction (P3) + equivalence matching (P1)	Accuracy
RCA	LLM-as-a-Judge (P2)	Average Score
ARC	LLM-based equivalence matching (P1)	Accuracy

Appendix

Table 3: Evaluation setup

Dataset	Base Model	AutoAdapt	AutoMLAgent	MLCopilot	HF Defaults	DS-Agent
UC Berkeley MATH	Phi-4-mini-Instruct	$27.8\pm 3.5$	$26.8\pm 2.4$	$27.2\pm 2.3$	$26.2\pm 0.06$	$\mathbf{30.0\pm 3.7}$
MedQA	Phi-4-mini-Instruct	$\mathbf{61.27\pm 1.1}$	$58.84\pm 1.3$	$57.50\pm 1.2$	$58.13\pm 0.04$	$59.86\pm 1.0$
CaseHold	Llama3.2-3B	$\mathbf{89.22\pm 0.9}$	$84.72\pm 1.1$	$82.63\pm 1.0$	$86.21\pm 0.08$	$87.28\pm 0.9$
PEM	Qwen2.5-Math-7B-Instruct	$\mathbf{67.69\pm 1.3}$	$64.53\pm 1.4$	$63.71\pm 1.5$	$65.78\pm 0.02$	$63.40\pm 1.6$
RCA	Phi-4	$\mathbf{2.30\pm 0.76}$	$2.23\pm 0.81$	$2.18\pm 0.77$	$2.23\pm 0.81$	$2.25\pm 0.66$
ARC	Llama3.1-8B-Instruct	$\mathbf{83.79\pm 1.2}$	$77.39\pm 1.1$	$77.56\pm 1.0$	$78.24\pm 0.03$	$77.47\pm 1.4$

Appendix

Table 4: Performance comparison across datasets and methods in the template aware (TA) setting.

Dataset	AutoAdapt	AutoMLAgent	MLCopilot	DS-Agent
UC Berkeley MATH	$\textbf{17.00}\pm 3.1$	$8.60\pm 2.7$	$0.60\pm 0.1$	$0.00\pm 0.00$
MedQA	$63.55\pm 2.1$	$46.97\pm 1.8$	$38.10\pm 1.2$	$\textbf{74.63}\pm 2.1$
CaseHold	$\textbf{92.53}\pm 1.4$	$85.90\pm 1.2$	$3.10\pm 0.8$	$3.16\pm 0.8$
PEM	$\textbf{65.14}\pm 2.3$	$28.03\pm 1.9$	$0.00\pm 0.00$	$0.00\pm 0.00$
RCA	$\textbf{2.30}\pm 0.76$	$1.44\pm 0.64$	$1.13\pm 0.53$	$1.23\pm 0.72$
ARC	$\textbf{90.87}\pm 2.2$	$76.71\pm 1.5$	$28.84\pm 1.2$	$14.85\pm 0.9$

Appendix

Table 5: Performance comparison across datasets and methods in the template free (TF) setting

Dataset	SR ( $1/(1+\text{iterations})$ )				NPS
	AutoAdapt	AutoMLAgent	MLCopilot	DS-Agent	AutoAdapt	AutoMLAgent	MLCopilot	DS-Agent
UC Berkeley MATH	1.00	0.11	0.17	1.00	0.17	0.09	0.01	0.00
MedQA	1.00	0.11	0.14	1.00	0.64	0.47	0.38	0.75
CaseHold	1.00	0.11	0.20	0.25	0.93	0.86	0.03	0.03
PEM	1.00	0.11	0.17	0.25	0.65	0.28	0.00	0.00
RCA	1.00	0.11	0.17	1.00	0.46	0.29	0.23	0.25

Appendix

Table 6: SR and NPS across datasets and baselines

Dataset	AutoAdapt	AutoMLAgent	MLCopilot	DS-Agent
UC Berkeley MATH	0.58	0.10	0.09	0.50
MedQA	0.82	0.29	0.26	0.87
CaseHold	0.96	0.49	0.12	0.14
PEM	0.83	0.20	0.08	0.12
RCA	0.73	0.20	0.20	0.62
ARC	0.95	0.44	0.23	0.24

Appendix

Table 7: Cumulative Score (CS) across datasets and baselines

Baseline	Manual Fixes Across Iterations
MLCopilot	It 1 : Incorrect argument name: evaluation_strategy → eval_strategy . It 2 : Precision changed from fp16 to bf16 . It 3 : Gradient check-pointing error. It 4 : Tokenizer padding added. It 5 : compute_metrics function updated. It 6 : Evaluation metric modified.
AutoML-Agent	It 1 : LoraConfig imported from peft , not trl . It 2 : Model name corrected. It 3 : LoRA-trained model merged and unloaded before saving. It 4 : AutoModelForCausalLM used instead of Seq2SeqLM . It 5 : lora_alpha used instead of alpha . It 6 : SFTTrainer.init() received unexpected argument tokenizer . Removed passing the tokenizer to SFTTrainer. It 7 : PeftModel loading corrected. It 8 : TrainingArgs incorrectly constructed and passed.
DS-Agent	It 1 : Incorrect parameter max_seq_length passed to SFTTrainer. It 2 : Model name incomplete. It 3 : peft_config incorrectly passed directly to SFTConfig instead of loading PEFT model.

Appendix

Table 8: Manual Corrections Required Across Iterations for Each Baseline

Domain Adaptation for LLMs

Domain adaptation for large language models tailors their knowledge and behavior to specialized fields, languages, and tasks.

See all 93 papers

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility
Oct 7, 2026Josh McGiff, Salma Mekaoui, Robert Shanahan +1Domain Adaptation for LLMs
University of Limerick, Ireland
LLM Agents for Automated Post-Training via Structured Recipe Optimization
Oct 13, 2026Ruohong Liu, Bohan Tang, Lekang Jiang +2LLM Post-Training Large Language Model-Guided Optimization
LIGHTSPEED · University of Cambridge · Independent Researcher
TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration
Apr 15, 2026Zerun Ma, Guoqiang Wang, Xinchen Xie +7LLM Fine-Tuning LLM Agent Training
Shanghai AI Laboratory · Fudan University

Oct 7, 2026J/K move · Enter open · S save

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

Josh McGiff, Salma Mekaoui, Robert Shanahan, Nikola S. Nikolov

University of Limerick, Ireland

Domain Adaptation for LLMs

Open paper AskPDF arXiv

Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.

Abstract

Domain Adaptation for LLMs

Explore similar work

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

LLM Agents for Automated Post-Training via Structured Recipe Optimization

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration