UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

Ye Chen; Weining Zhang

cs.AIAug 21, 2026

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

Authors: Ye Chen, Weining Zhang

Organizations: Alibaba Group · Cheung Kong Graduate School of Business

Abstract

Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.

Figures & tables

Checkpoint	Released	Hidden	Layers	Heads (Q/KV)	FFN	Vocab	Hop to next
Qwen1.5-7B-Chat	2024-02	4096	32	32 / 32 (MHA)	11008	151936	incompatible
Qwen2-7B-Instruct	2024-06	3584	28	28 / 4 (GQA)	18944	152064	compatible
Qwen2.5-7B-Instruct	2024-09	3584	28	28 / 4 (GQA)	18944	152064	incompatible
Qwen3-8B	2025-04	4096	36	32 / 8 (GQA)	12288	151936	—
Continuation release : Qwen2.5-7B-Instruct-1M (2025-01) — identical to Qwen2.5-7B-Instruct in every
dimension above, and a continued-pretraining descendant of the same weights (long-context extension).

Table 1: The measured lineage, from the released checkpoints’ config.json . Shape compatibility governs whether weight-space transfer (adapter copying, diff-vector porting) is defined at all : it requires identical hidden width, depth, and attention geometry. Exactly one of the three hops qualifies.

Task	Class	Train / Val / Test	Metric	Source
Banking77	D (77-way intent)	9,003 / 1,000 / 3,080	accuracy	Casanueva et al. (2020)
CLINC150	D (151-way intent, incl. OOS)	10,000 / 1,000 / 5,500	accuracy	Larson et al. (2019)
Spider	S (text-to-SQL)	6,500 / 500 / 1,034 (dev)	execution acc.	Yu et al. (2018)
FinQA	S (numerical programs)	6,251 / 500 / 1,147	program acc.	Chen et al. (2021)
xLAM-FC	A (function calling)	10,000 / 500 / 2,000 (held-out)	call-set match	Liu et al. (2024)
glaive-FC	A (calling + refusal)	7,580 / 505 / 2,000 (group re-split)	call-set match	glaive-v2 corpus

Table 2: Task suite. D = discriminative, S = structured generation, A = agentic tool use. Licenses as listed by official distributions.

ID	Strategy	Data need	Description	Measured
Freeze	keep old specialist	—	serve $s_{b_{i},t}$ unchanged	9 hops
Adopt-0	new base, zero-shot	—	prompt-only cheap adoption	4 gens
Adopt-k	new base, 5-shot	—	fixed exemplars	4 gens (D)
Copy / PortDiff	weight-space port	none	load $s_{b_{i},t}$ ’s LoRA on $b_{j}$ ( Lin et al., 2025 )	2 $\to$ 2.5: 6 tasks (+3 at 1.8B); 2.5 $\to$ 1M: 3 tasks
Refresh-D	distillation refresh	unlabeled inputs	old specialist labels retained inputs; fresh LoRA on $b_{j}$ (relabel variant of Wang et al., 2024 )	2.5 $\to$ 3, 3 tasks
Retrain	full QLoRA on $b_{j}$	full labels	reference denominator of Eq. 1	all generations, 6 tasks, 2 scales

Table 3: Strategy conditions and measurement status. For LoRA specialists, pasting the adapter and adding its merged

\Delta W=BA

to the new base are the same operation, so the Copy measurement covers diff-vector porting as well.

Panel	Adopt/Freeze	Copy	Refresh-D	Retrain- $N$	Reference
B77/Spider/xLAM, 7–8B	4 gens + 1M	2 $\to$ 2.5, 2.5 $\to$ 1M	2.5 $\to$ 3 (3 seeds)	Spider only a	4 gens + 1M (3 seeds)
CLINC/FinQA/glaive, 7–8B	3 gens	2 $\to$ 2.5	— b	—	3 gens
B77/Spider/xLAM, 1.5–1.8B	4 gens	s2 $\to$ s2.5	—	—	4 gens
B77/CLINC, OLMo	7 checkpoints	5 documented edges	—	—	7 checkpoints
Shape-breaking hops	(= Adopt/Freeze)	undefined c	—	—	(= Reference)

Table 4: Coverage matrix: which actions are measured on which edges. The design is deliberately sparse, not factorial; empty cells are stated, not implied. a Label ladder run on the highest-coupling task only. b Refresh-D requires retained inputs; measured on the newest hop of the original three tasks. c Learned heterogeneous mappings and distillation-transfer baselines are extension points of the released protocol, not measured here.

Figure 1: Durability in real release time (x-axis: months since the gen-0 specialists were trained; all points measured,

n

= 3,080 / 1,034 / 2,000 test items). (a) Frozen gen-0 specialists (horizontal lines) against each task’s rising adoption floor. (b) The same data as advantage over the floor, with half-value (

H_{50}

, dotted) and crossover (

H_{0}

) horizons marked; horizons are interval-censored by the discrete release calendar. Generated from the released study registry.

	Qwen1.5-7B		Qwen2-7B		Qwen2.5-7B		Qwen3-8B
Task	0-shot	Spec.	0-shot	Spec.	0-shot	Spec.	0-shot	Spec.
Banking77	.4354	.9302	.5747	.9276	.6071	.9312	.6438	.9321
(5-shot)	.5679		.6273		.6357		.6594
Spider	.5242	.6963	.6567	.6992	.6760	.7321	.6963	.7582
xLAM-FC	.5770	.8205	.6315	.8225	.7940	.8480	.7700	.8455

Table 5: Lineage scores (test sets; greedy decoding; 4-bit inference). “Spec.” = specialist fine-tuned on that generation (the retrain reference for hops ending there). Bootstrap 95% CIs for all cells are released; typical half-widths

\pm

0.9pp (Banking77),

\pm

2.8pp (Spider),

\pm

1.6pp (xLAM).

	$R(\textsc{Freeze})$				$R(\textsc{Copy})$	Refresh-D
Task	1.5 $\to$ 2	2 $\to$ 2.5	2.5 $\to$ 3	mean	2 $\to$ 2.5	$R$	cost (min)
Banking77	1.007	0.989	0.997	0.998	–0.551	0.997	39
Spider	0.932	0.414	0.578	0.641	0.139	1.047	255
xLAM-FC	0.990	0.528	1.033	0.850	0.370	0.960	331

Table 6: Retention by strategy. Freeze is measured on all nine hops; Copy on the one shape-compatible hop (2

\to

2.5); Refresh-D on the most recent hop (2.5

\to

3), with cost = teacher relabeling + student training on one RTX 4090.

R<0

: worse than not transferring.

R>1

: beats gold-label retraining.

Task	Class	Qwen2	Qwen2.5	Qwen3	$\Delta$ (pp)	Evidence
Spider 3s	S	.7025 [.695–.713]	.7321 [.730–.734]	.7501 [.740–.758]	$+4.8$	all seeds ordered
xLAM-FC 3s	A	.8225	.8473 [.846–.848]	.8460 [.843–.850]	$+2.3$	$3{\times}10^{-4}$
FinQA	S	.5981	.5990	.6103	$+1.2$	$0.22$
Banking77	D	.9276	.9312	.9321	$+0.5$	$0.16$
glaive-FC	A	.9640	.9650	.9645	$+0.05$	$1.0$
CLINC150 3s	D	.9313 [.927–.934]	.9322	.9262 [.918–.931]	$-0.5$	within seed noise

Table 7: Fine-tuned ceilings across the three most recent 7–8B generations (specialist scores; reference for hops ending at each generation).

\Delta

= Qwen3 minus Qwen2;

p

from the exact McNemar test on the paired test set.

Figure 2: (a) Retention of naive adapter copying between architecturally identical checkpoints, in two families. Between independent pretraining runs (Qwen2

\to

2.5, nine task/scale cells; OLMo-1

\to

1.7, two cells) copying lands at or below the no-adapter floor; between checkpoints connected by continued pretraining (Qwen2.5

\to

2.5-1M, three tasks; OLMo-2 stage-1 237k

\to

248k , two cells) it retains 0.82–1.45 (Spider’s 1.45 reflects a small denominator: the 1M reference gains only 3.5pp over its floor); and once the continuation runs long ( 237k

\to

main,

{\sim}

2.9T tokens plus anneal and soup) it collapses below the floor—Table 8 ’s distance ablation shows the 2.9T-token distance alone suffices, with the release operations adding nothing further. (b) The pattern replicates at 1.5–1.8B: the Banking77 ceiling stays flat while zero-shot climbs 22.6pp, the Spider ceiling rises with every generation, and the generation-0 Spider specialist is overtaken within the lineage.

Target edge	Task	Copy	Floor	Ref	$R$	$p$ vs floor
OLMo-1 $\to$ 1.7 (fresh run)	B77	.0266	.2464	.9292	$-0.32$	$<\!10^{-169}$ ( $\downarrow$ )
	CLINC	.2364	.2482	.9151	$-0.02$	$0.063$ ( $\approx$ )
237k $\to$ 248k (+46B cont.)	B77	.8179	.0097	.9305	$+0.88$	$<\!10^{-300}$
	CLINC	.8951	.0120	.9078	$+0.99$	$<\!10^{-300}$
237k $\to$ 928k (+2.9T cont.)	B77	.1279	.1432	.9289	$-0.02$	$0.074$ ( $\approx$ )
	CLINC	.1433	.0315	.9204	$+0.13$	$<\!10^{-138}$

Table 8: OLMo panel: copying the same intent adapters along a fully documented training trajectory. The top pair is a verified fresh pretraining run; the bottom four rows per task move the target along OLMo-2’s own trajectory—a 46B-token continuation, a 2.9T-token continuation (end of stage 1, before any annealing), the stage-2 anneal (before souping), and the released soup. Copy vs. floor and vs. reference tested with exact McNemar (

\downarrow

: Copy significantly below the floor;

\approx

: indistinguishable). CLINC150 uses the lenient metric (strict released per cell).

Figure 3: Copying one adapter (trained at OLMo-2 stage-1 step 237k) onto four later checkpoints of the same documented run. Retention falls to the no-adapter floor within the pure-continuation segment; the anneal and soup edges change the copied score by at most 1.4pp (the lower

R

reflects their higher floors). Generated from the released records.

Transition	NF%	PF%	Net (pp)
Banking77: 2.5-specialist $\to$ 3-reference	1.23	1.33	$+0.10$
Spider: 2.5-specialist $\to$ 3-reference	4.74	7.35	$+2.61$
xLAM: 2.5-specialist $\to$ 3-reference	3.90	3.65	$-0.25$
CLINC150: 2-specialist $\to$ 3-specialist	2.55	1.65	$-0.89$
FinQA: 2-specialist $\to$ 3-specialist	4.27	5.49	$+1.22$
glaive-FC: 2-specialist $\to$ 3-specialist	0.50	0.40	$-0.10$

Table 9: Negative/positive flip rates between strategy pairs on the fixed test sets (per-example records; NF = old correct, new wrong).

Policy	Mean regret	Max regret	GPU-h	Gold-label eps.	Regressions
Always- Freeze	1.80pp	11.5pp	0	0	0
Always- Adopt	32.35pp	91.6pp	0	0	33
Always-retrain	0.21pp	2.0pp	47.7	33	1
Copy-if-shape	17.11pp	90.8pp	25.8	15	14
Proposed	0.37pp	6.6pp	15.6	9	0

Table 10: Offline policy replay over 33 measured upgrade episodes. Regret is against the best measured action per episode; a regression episode is one where the chosen action falls more than

\varepsilon

below Freeze .

$n$	false-open (15 null cells)		false-equivalence
	point-estimate gate	three-zone + collect	(6 upgrade cells)
100	17.9%	1.9%	0.5%
200	15.2%	2.1%	0.2%
300	13.0%	2.0%	0.2%
600	8.2%	1.8%	0.1%
1000	1.8%	1.5%	0.0%

Table 11: Small-evidence operating curve, replayed on real records. At enterprise gate sizes the point-estimate gate opens the upgrade waterfall on one in six ground-truth-null episodes; the three-zone gate holds the false-open rate near its nominal level at every

n

, and its “equivalence” verdicts are almost never wrong. The cost of that conservatism is bounded and we state it: mean serving-accuracy regret 0.58pp vs 0.48pp at

n{=}100

(the point-estimate rule’s spurious opens occasionally land on real upgrades; its migration and regression costs are outside this metric).

labeling budget $b$	i.i.d. sampling	disagreement-first
25	0%	37%
50	4%	48%
100	13%	67%

Table 12: Probability that the upgrade direction settles (exact sign test, correct sign) under a fixed labeling budget, averaged over the cells with

|\delta_{\text{full}}|\geq 1

pp. Labeling

b

random items wastes almost the entire budget on items both systems already get right; labeling

b

disagreements concentrates it on the only items that carry decision information.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 755 papers

MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale
Aug 11, 2026Yuhang Yao, Zeyu Wang, Wanyi Chen +8Continual Learning for LLM Agents LLM Fine-Tuning
Carnegie Mellon University · University of California, Los Angeles · Soochow University +3
MSA-CITE: A Co-Adapted LoRA Specialist Ecology for Fixed-Budget Small-Model Inference
Aug 11, 2026Ruitong Li, Binjie Guo, Aisheng Mo +3Language Model Ensembles Efficient Language Model Inference
The University of Hong Kong · Zhejiang University · Independent Researcher
ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement
Oct 5, 2026Xingbo Yao, Xiaoman Wang, Zhengwu Lei +11LLM Fine-Tuning LLM Post-Training
HKUST(GZ) · Xiaohongshu Inc. · ECNU +3

Aug 11, 2026J/K move · Enter open · S save

MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale

Yuhang Yao, Zeyu Wang, Wanyi Chen, Tongyun Yang+7

Carnegie Mellon University · University of California, Los Angeles · Soochow University+3

Continual Learning for LLM Agents LLM Fine-Tuning LLM Agent Training LLM Routing Knowledge Distillation

Open paper AskPDF arXiv

LLM agents execute heterogeneous sequences of model calls within a single task: some invocations require careful reasoning, while others are structured steps such as formatting or tool-argument construction. Prior routing methods exploit this asymmetry by assigning easy invocations to a cheaper small model and difficult ones to a large model. Such policies reduce inference cost, but they leave the small model's capability unchanged, so attainable savings remain bounded by the work the student can already solve. MERA instead improves the small model itself, using a single model invocation as the unit of adaptation. In each cycle, MERA replays failed student invocations to obtain execution-verified teacher demonstrations, distills recurring procedures into an iteratively updated SkillBook, and fine-tunes a student LoRA adapter via supervised learning and optional GRPO. Routing serves as supporting machinery for deployment: the improved student is served behind a cost-calibrated router with verifier-backed fallback, and a candidate SkillBook, adapter, or router is admitted only when joint replay preserves task quality. Empirically, four-cycle adaptation raises Qwen2.5-Coder-1.5B from 28.7% to 49.7% pass on held-out HumanEval+MBPP. Under verifier-backed fallback, the deployed policy retains 88.3% pass at 60.8% of always-Luna cost. On TAU-2, a fine-tuned Qwen3.5-2B improves from 14/35 to 18/35 and matches an unadapted 4B model. These results indicate that verifier-backed multi-cycle adaptation can increase small-model capability, rather than only routing around a fixed student.

Abstract

LLM Fine-Tuning

Explore similar work

MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale

MSA-CITE: A Co-Adapted LoRA Specialist Ecology for Fixed-Budget Small-Model Inference

ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement