HuatuoGPT-3: RL-Only Domain Adaptation from Base Models

Junying Chen; Xinyuan Xie; Ziniu Li; Wenyuan Gu; Jianquan Li; Xiang Wan; Guangjun Yu; Ruoyu Sun

cs.CLOct 5, 2026

HuatuoGPT-3: RL-Only Domain Adaptation from Base Models

Authors: Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li, Xiang Wan, Guangjun Yu, Ruoyu Sun, +2 more

Organizations: The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · Shenzhen Loop Area Institute · National Health Data Institute, Shenzhen

Abstract

Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale OnePO to produce HuatuoGPT-3, an open-source medical LLM series whose 27B variant reaches 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, surpassing frontier models such as GPT-6 Astra. Models and code are available at https://github.com/FreedomIntelligence/HuatuoGPT-3.

Figures & tables

Group	Method	External (Teacher) Guidance	Cold Start	Beyond Imitation	Late Imitation Dependence	Output Diversity
w/ SFT	SFT [ 1 , 2 , 3 , 4 ]	teacher outputs (SFT)	easy	✗	high	low
	SFT+RL [ 7 , 10 ]	teacher outputs (SFT)	easy	✓	low	medium
RL-only	Pure On-Policy RL [ 22 ]	none	hard	✓	low	high
	Standard Mixed-Policy RL [ 21 ]	persistent teacher outputs	easy	✓	high	medium
	OnePO (Ours)	transient teacher outputs	easy	✓	low	high

Table 1: Conceptual comparison of domain adaptation paradigms. SFT-based methods enable an easy cold start via imitation but reduce later diversity. RL-only methods better preserve exploration. Among them, OnePO stands out: it uses teacher outputs as transient guidance and retires them once the policy surpasses them, achieving an easy cold start without sacrificing later autonomy.

Figure 1 : Results of two pilot studies. Left: AHA-Medicine study examining whether each route can absorb a synthetic fact introduced through teacher outputs and reason beyond imitation under a zero-data-leakage setting. Reward $=0.5$ indicates learning the fact, and reward $=1.0$ indicates recognizing it as fictitious. Right: The anchoring pilot study examining whether persistent teacher outputs limit later-stage RL improvement. Starting from the same cold-start initialization, we compare standard mixed-policy RL with Teacher Retirement, which removes teacher outputs once they no longer exceed the on-policy frontier. See Appendix C for details.

Figure 2 : Overview of OnePO . Top: the training dynamics, where teacher outputs accelerate domain knowledge acquisition early on, and optimization gradually returns to standard on-policy exploration as training progresses. Bottom: one iteration of OnePO training, where teacher outputs are first filtered by Teacher Retirement and the retained ones then enter advantage computation and policy update together with on-policy outputs.

Figure 3 : Illustration of the two data types and reward mechanisms. Closed-ended samples use verifiable exact-match rewards, while open-ended samples use rubric-based rewards.

	Open-ended			Closed-ended
Model	HealthBench Professional $\bigstar$	HealthBench (Total)	HealthBench (Hard)	Medbullets (5-option)	MMLU-Pro (Med)	MedXpertQA (Text)
Representative Baselines
GPT-6 Astra (high)	62.5	57.8	32.1	88.9	87.6	64.4
GPT-5.2 (high)	52.8	59.9	40.5	88.9	90.0	54.2
Gemini 3.8 Flash	49.9	54.3	26.4	91.9	88.4	63.0
DeepSeek-V4.1-Flash (high)	48.5	57.7	32.7	86.0	85.8	51.6

Table 2 : Main results on open-ended medical response quality and closed-ended medical knowledge and reasoning.

\bigstar

marks HealthBench Professional, evaluated using its official grader (GPT-5.4 with low reasoning effort) and length adjustment, while HealthBench Total/Hard use GPT-4.1 without length adjustment . Green gains indicate OnePO improvements over Pure RL with the same base model and RL hyperparameters. SFT+RL and OnePO use the same offline teacher outputs for a fair comparison. The HuatuoGPT-3 series comprises medical LLMs trained with OnePO on expanded datasets across different backbones and model sizes.

Figure 4 : HealthBench Professional performance comparison. HuatuoGPT-3 models achieve higher performance with fewer parameters and lie above the scaling trend of the baseline models.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Open-ended		Closed-ended
Setting	HealthBench (Total)	HealthBench (Hard)	Medbullets	MMLU-Pro	MedXpertQA
Qwen3-8B-Base	22.1	0.0	30.7	41.2	11.7
w/ Verifiable Only	24.2	3.2	64.5	80.5	24.8
w/ Rubric Only	63.7	36.6	44.8	67.0	12.5
w/ Mixed (Ours)	65.4	39.1	64.0	81.2	24.9

Appendix

Table 7 : Ablation of training data composition. Verifiable Only : use only 10K closed-ended multiple-choice data and verifiable reward. Rubric Only : use only 10K open-ended clinical tasks and rubric-based reward. Mixed (Ours) : use the full 20K data and jointly use the two rewards. The mixed setting achieves the best overall trade-off performance, indicating that dense rubric supervision and sparse verifiable signal are complementary.

		Open-ended (HealthBench)		Closed-ended
Teacher Source	Setting	Total	Hard	Medbullets	MMLU-Pro(Med)	MedXpertQA
None	Pure RL	59.8	25.2	48.1	75.0	20.0
GPT-5 Chat	SFT	30.4	4.8	49.7	72.5	21.7
	SFT+RL	63.6	37.3	61.9	78.2	21.5
	OnePO	65.4	39.1	64.0	81.2	24.9
DeepSeek-V3.2 (thinking)	SFT	36.8	2.4	56.6	71.0	17.2

Appendix

Table 8 : Ablation results of different training settings (Pure RL / SFT / SFT+RL / OnePO) under different teacher sources.

Floor $c$	HealthBench	Medbullets	MMLU-Pro (Med)
0.00	49.1	49.7	73.1
0.01	58.3	59.7	77.2
0.05	64.8	63.2	79.5
0.10	65.4	64.0	81.2
0.20	62.1	61.5	79.8
0.50	60.2	59.1	77.3

Appendix

Table 9 : Ablation of the probability floor

c

. All runs use Qwen3-8B-Base, GPT-5 Chat as the teacher source, and the same OnePO training setup.

Method	Steps to reward $\geq 0.5$
OnePO	7
OnePO w/ KL penalty	7
OnePO w/ KL loss	8
OnePO w/ KL loss + KL penalty	8
OnePO w/o probability floor	Fail
OnePO w/o rescaling	Fail

Appendix

Table 10 : KL-constrained AHA-Medicine pilot. “Fail” means the model does not absorb the target knowledge within 100 RL steps. The KL loss coefficient is

0.001

, and the KL reward-penalty coefficient is

0.01

Method	S1	S2	S3	S4	S5	S6	S7	S8
OnePO	-6.9735	-6.1488	-4.2228	-2.7612	-1.0261	-0.2838	-0.0156	-0.0122
OnePO w/ KL loss + KL penalty	-6.9735	-6.6096	-5.5103	-3.5550	-2.2576	-1.0866	-0.3733	-0.0406
OnePO w/o rescaling	-6.9735	-7.1113	-7.1156	-7.1270	-6.9345	-6.9741	-6.5402	-6.6415

Appendix

Table 11 : Average log-probability of the target “AHA-Medicine” span across the first eight training steps.

Method	S1	S2	S3	S4	S5	S6	S7	S8	Avg.
OnePO	0.0000	0.0645	0.0608	0.0280	0.0038	0.0047	0.0072	0.0064	0.0219
OnePO w/ KL loss + KL penalty	0.0000	0.0410	0.0238	0.0072	-0.0125	0.0015	0.0033	0.0018	0.0083
OnePO w/o rescaling	0.0000	0.0533	0.0590	0.0755	0.1160	0.1456	0.1471	0.1779	0.0968

Appendix

Table 12 : Rollout-based empirical KL estimates in the AHA-Medicine pilot. Small negative values may appear due to sampling noise.

Method	Writing		Law
Method	WritingBench	CreativeWriting-v3	LexEval	LawBench
Qwen3-4B-Base	33.7	23.9	18.6	41.8
w/ Pure RL	67.4	41.1	46.5	60.8
w/ SFT+RL	71.2	45.6	50.7	64.5
w/ OnePO	74.6	47.8	51.9	67.4

Appendix

Table 13 : Cross-domain validation on writing and law using Qwen3-4B-Base. OnePO improves over SFT+RL under the same teacher source in both domains.

Metric vs. Human Label	8B Grader (Training)	GPT-4.1 Grader (Evaluation)
F1 score	0.81	0.85
Cohen’s kappa	0.62	0.72

Appendix

Table 14 : Agreement between graders and doctor annotations on 200 query-answer-rubric-criterion instances.

Model	8B Grader	GPT-4.1 Official	GPT-5 Chat Grader
Qwen3-8B	48.5	45.9	41.8
Qwen3-8B w/ OnePO	70.7	67.2	65.1
Gain	+22.2	+21.3	+23.3

Appendix

Table 15 : HealthBench re-evaluation with different graders.

Train step	0	20	40	60	80	100	120
Open-ended reward	0.34	0.69	0.69	0.70	0.70	0.75	0.77
Closed-ended reward	0.29	0.38	0.48	0.52	0.55	0.58	0.59

Appendix

Table 16 : Training reward curves for open-ended rubric-based rewards and closed-ended verifiable rewards.

Explore similar work

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation
Aug 31, 2026Kwangmin Ki, Yunhun Nam, Jongheon Jeong +1Large Language Model Adaptation Domain Adaptation
Korea University · Yonsei University
Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA
Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3Large Language Model Adaptation Domain Adaptation
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
May 28, 2026Runze Xu, Arpit Garg, Hemanth Saratchandran +1Large Language Model Adaptation Rapid Adaptation
Australian Institute for Machine Learning Adelaide University Adelaide SA 5000

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Kwangmin Ki, Yunhun Nam, Jongheon Jeong, Jaehyung Kim

Korea University · Yonsei University

Large Language Model Adaptation Domain Adaptation Catastrophic Forgetting Domain-Specific Language

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.

Abstract

Explore similar work

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting