Mashup Learning: Faster Finetuning by Remixing Past Checkpoints

Sofia Maria Lo Cicero Vaina; Artem Chumachenko; Max Ryabinin

cs.LGMar 10, 2026

Mashup Learning: Faster Finetuning by Remixing Past Checkpoints

Authors: Sofia Maria Lo Cicero Vaina, Artem Chumachenko, Max Ryabinin

Organizations: Together AI.

Abstract

Finetuning on domain-specific data is a well-established method for enhancing LLM performance on downstream tasks. Training on each dataset produces a new set of model weights, resulting in a multitude of checkpoints saved in-house or on open-source platforms. However, these training artifacts are rarely reused for subsequent experiments despite containing improved model abilities for potentially similar tasks. In this paper, we propose Mashup Learning, a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks. Our procedure identifies the most relevant historical checkpoints for a target dataset, aggregates them with model merging, and uses the result as an improved initialization for training. Across 8 standard LLM benchmarks, four models, and two collections of source checkpoints, Mashup Learning consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch. It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy, including all selection and merging overhead.

Figures & tables

Figure 1: A schematic example of Mashup Learning.

Dataset	Short name	Train size	Val size
ARC-Easy	ARC-e	2,251	570
CommonsenseQA	CSQA	9,741	1,221
HellaSwag	Hella.	39,905	10,042
MathQA	MathQA	29,837	4,475
OpenBookQA	OBQA	4,957	500
PIQA	PIQA	16,113	1,838

Table 1: Dataset summary for our primary experiments.

Method	ARC-e	Hella.	MathQA	OBQA	PIQA	Wino.	Avg.
From scratch	84.0	93.2	21.7	74.6	85.2	81.4	73.4
Text-to-LoRA	82.4	47.6	26.3	71.2	80.2	66.1	62.3
Text-to-LoRA used as init	83.1	91.6	26.2	74.6	85.1	74.2	72.5
Mashup Learning at init	85.0	61.2	26.4	77.9	80.6	64.9	66.0
Mashup Learning	87.9	95.1	32.1	86.6	86.5	82.5	78.5

Table 2: Finetuning methods comparison on Mistral-7B-Instruct-v0.2, Lots-of-LoRAs setup. Accuracy results across six benchmarks comparing different LoRA merging and initialization approaches. The best result is highlighted in bold.

Model	Setup	Method	ARC-e	CSQA	Hella.	MathQA	OBQA	PIQA	SIQA	Wino.	Avg.
Gemma-3 1B	LoRA	From scratch	75.0	71.9	81.4	39.3	70.3	77.0	74.2	70.6	70.0
	LoRA	Mashup Learning	78.8	73.5	82.7	39.8	75.6	78.4	74.1	71.2	71.8
	Full FT	From scratch	73.9	68.8	79.8	37.3	68.3	76.1	73.0	69.2	68.3
	Full FT	Mashup Learning	78.1	70.5	80.2	39.1	73.1	77.7	73.3	69.6	70.2
Gemma-2 2B	LoRA	From scratch	90.1	82.5	93.5	44.7	85.0	85.3	81.5	83.6	80.8
	LoRA	Mashup Learning	91.6	82.7	93.6	45.8	85.5	86.3	81.7	84.4	81.5

Table 3: Mashup Learning vs. training from scratch, leave-one-out setup. Average accuracy results across eight benchmarks over three seeds. The best result for each model is highlighted in bold.

Model	Setup	Method	ARC-e	CSQA	Hella.	MathQA	OBQA	PIQA	SIQA	Wino.	Avg.
Gemma-3 1B	LoRA	From scratch	74.1	80.3	73.9	88.8	82.7	73.0	78.5	84.1	79.4
	LoRA	Mashup	28.9	48.5	61.1	—	43.1	47.4	75.0	69.1	59.1
	Full FT	From scratch	62.8	65.5	67.8	72.4	74.2	60.3	70.8	64.8	67.3
	Full FT	Mashup	9.6	53.5	61.0	55.5	41.0	42.1	51.4	—	51.8
Gemma-2 2B	LoRA	From scratch	53.7	72.2	65.0	80.8	79.7	66.1	71.2	72.5	70.2
	LoRA	Mashup	11.7	69.4	58.1	74.8	64.9	55.8	73.5	64.0	59.0

Table 4: Convergence speedup across all models. Percentage of training steps at which each method reached 99% of converged from-scratch accuracy. Lower is better. “—” if any seed did not reach 99% of from-scratch accuracy.

Model	Setup		ARC-e	CSQA	Hella.	MathQA	OBQA	PIQA	SIQA	Wino.	Avg.
Gemma-3 1B	LoRA	From scratch (s)	101	162	611	356	134	261	365	400	299
		Mashup (s)	90	146	438	337	99	181	400	356	256
		Ratio (%)	89%	90%	72%	95%	73%	69%	109%	89%	86%
	Full FT	From scratch (s)	298	308	2799	1801	231	842	1026	1021	1041
		Mashup (s)	127	188	2001	1177	136	398	656	991	709
		Ratio (%)	42%	61%	72%	65%	59%	47%	64%	97%	63%

Table 5: Wall-clock training time across all models. Training time in seconds at the best learning rate. Mashup includes relevance estimation and merge overhead, with training time scaled by convergence percentage. Ratio

<100\%

means Mashup is faster.

Method	Checkpoint Selection	ARC-e	CSQA	Hella.	MathQA	OBQA	PIQA	SIQA	Wino.	Avg.
LoRA	–	87.32	83.06	93.38	21.27	84.38	86.90	82.79	84.62	77.96
Mashup	Random	88.42	82.73	93.63	21.16	85.42	87.77	32.94	84.54	72.08
	By accuracy	88.24	83.47	93.44	40.38	85.62	87.61	80.99	84.46	80.53
	By loss	88.24	83.47	93.44	21.54	85.62	86.95	81.71	84.46	78.18
	Oracle	89.52	84.79	94.02	42.18	87.50	88.65	82.63	84.94	81.78

Table 6: Relevance Methods Comparison. Accuracy results across multiple benchmarks comparing different relevance selection methods. The best result is highlighted in bold and the second best is underscored.

Figure 2: Accuracy on PIQA as a function of the number of samples used for checkpoint selection, evaluated on Mistral-7B-Instruct-v0.2.

Figure 3: Mean rank by accuracy of each combination of merging method and number of merged models across 8 benchmarks (ARC-Easy, CommonsenseQA, HellaSwag, MathQA, OpenBookQA, PIQA, SocialIQA, Winogrande) in a leave-one-out setup for Mistral-7B-Instruct-v0.2.

Figure 4: Gemma-3 4B LoRA sensitivity of training results to learning rates. Mean accuracy on 8 benchmarks across 3 seeds

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Score	Params
Gemma 3n E4B IT	1319	4B
Gemma 3 4B IT	1303	4B
Ministral 8B	1237	8B
Llama 3 8B Instruct	1224	8B
Llama 3.1 8B Instruct	1212	8B
Gemma 2 2B IT	1199	2B

Appendix

Table 7: Top 10 open-source models under 9B parameters on the LM Arena text leaderboard (February 2026).

Hyperparameter	LoRA	Full FT
Optimizer	AdamW	AdamW
Weight Decay	0.01	0.01
LR Schedule	Cosine	Cosine
LR Warmup	10%	10%
Batch Size	32	32
Epochs	1	1

Appendix

Table 8: Training hyperparameters for LoRA and full fine-tuning on Mistral-7B-Instruct-v0.2 and Gemma models.

Model	Setup	Method	ARC-e	CSQA	Hella.	MathQA	OBQA	PIQA	SIQA	Wino.
Gemma-3 1B	LoRA	From scratch	$2.7\text{e-}4$	$5\text{e-}4$	$2.7\text{e-}4$	$2.7\text{e-}4$	$5\text{e-}4$	$3.8\text{e-}4$	$3.8\text{e-}4$	$3.8\text{e-}4$
	LoRA	Mashup Learning	$2.7\text{e-}4$	$3.8\text{e-}4$	$3.8\text{e-}4$	$2.7\text{e-}4$	$3.8\text{e-}4$	$1.6\text{e-}4$	$5\text{e-}4$	$3.8\text{e-}4$
	Full FT	From scratch	$1.6\text{e-}5$	$2.7\text{e-}5$	$2.7\text{e-}5$	$1.6\text{e-}5$	$2.7\text{e-}5$	$2.7\text{e-}5$	$2.7\text{e-}5$	$2.7\text{e-}5$
	Full FT	Mashup Learning	$2.7\text{e-}5$	$2.7\text{e-}5$	$2.7\text{e-}5$	$1.6\text{e-}5$	$2.7\text{e-}5$	$1.6\text{e-}5$	$1.6\text{e-}5$	$2.7\text{e-}5$
Gemma-2 2B	LoRA	From scratch	$2.7\text{e-}4$	$3.8\text{e-}4$	$1.6\text{e-}4$	$1.6\text{e-}4$	$3.8\text{e-}4$	$2.7\text{e-}4$	$1.6\text{e-}4$	$1.6\text{e-}4$
	LoRA	Mashup Learning	$3.8\text{e-}4$	$2.7\text{e-}4$	$1.6\text{e-}4$	$2.7\text{e-}4$	$3.8\text{e-}4$	$3.8\text{e-}4$	$1.6\text{e-}4$	$1.6\text{e-}4$

Appendix

Table 9: Selected learning rates. Best learning rate per model, setup, method, and task, chosen by maximizing mean validation accuracy across seeds. LoRA rates are swept over

[5\mathrm{e}{-5},\,5\mathrm{e}{-4}]

; full fine-tuning rates over

[5\mathrm{e}{-6},\,5\mathrm{e}{-5}]

Panel of Figure 5 — Figure 5: Gemma-3 1B: per-task learning rate sensitivity.

Panel of Figure 6 — Figure 6: Gemma-2 2B: per-task learning rate sensitivity.

Panel of Figure 7 — Figure 7: Gemma-3 4B: per-task learning rate sensitivity.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Apr 6, 2025

Exact Unlearning of Finetuning Data via Model Merging at Scale

Approximate unlearning has gained popularity as an approach to efficiently update an LLM so that it behaves (roughly) as if it was not trained on a subset of data to begin with. However, existing methods are brittle in practice and can easily be attacked to reveal supposedly unlearned information. To alleviate issues with approximate unlearning, we instead propose SIFT-Masks (SIgn-Fixed Tuning-Masks), an exact unlearning method based on model merging. SIFT-Masks addresses two key limitations of standard model merging: (1) merging a large number of tasks can severely harm utility; and (2) methods that boost utility by sharing extra information across tasks make exact unlearning prohibitively expensive. SIFT-Masks solves these issues by (1) applying local masks to recover task-specific performance; and (2) constraining finetuning to align with a global sign vector as a lightweight approach to determine masks independently before merging. Across four settings where we merge up to 500 models, SIFT-Masks improves accuracy by 5-80% over naive merging and uses up to 250x less compute for exact unlearning compared to other merging baselines.

Kevin Kuo, Amrith Setlur, Kartik Srinivas +2

Computer Science Department · Machine Learning Department

Machine Unlearning LLM Unlearning Language Model Merging

Ask PDF

May 7, 2026

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning

Large language models (LLMs) can learn new tasks through fine-tuning, but learning tasks sequentially often leads to catastrophic forgetting of earlier ones. This paper proposes CRAFT, a continual learning framework that keeps model weights frozen and instead learns low-rank interventions on hidden representations. Similar tasks share an intervention to help knowledge transfer, with similarity judged by how each task alters the model's output distribution rather than by a learned gating network. Within a shared intervention, a Kullback-Leibler (KL) penalty keeps each new task close to the intervention's existing behavior, which limits forgetting of earlier tasks, while a small, task-private intervention in the deepest layers preserves dedicated capacity that later tasks cannot overwrite. The learned interventions also make it possible to identify the task at test time, so task identity labels are not needed. Across multiple benchmarks and model scales, CRAFT improves overall performance and reduces forgetting compared with strong LoRA-based methods, while remaining robust to task ordering.

Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz +2

Iowa State University · Maro Systems, USA · University of California, Berkeley +1

Continual Learning for LLMs Low-Rank Adaptation Incremental Learning Catastrophic Forgetting

Ask PDF

May 26, 2026

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a Rank-1 Subspace phenomenon: while raw optimization steps oscillate violently, consecutive merged checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a river-valley landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose Extra-Merge, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.

Wenjie Zhou, Bohan Wang, Hongtao Zhang +3

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2

Language Model Pretraining Neural Network Optimization Weight-Space Analysis Language Model Merging

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

Exact Unlearning of Finetuning Data via Model Merging at Scale

CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training