Learning New Facts with QLoRA: An Acquisition-Retention Frontier

Estelle Zheng; Sébastien Warichet; Emmanuel Helbert; Christophe Cerisara

cs.CLAug 26, 2026

Learning New Facts with QLoRA: An Acquisition-Retention Frontier

Authors: Estelle Zheng, Sébastien Warichet, Emmanuel Helbert, Christophe Cerisara

Organizations: LORIA, CNRS, France · Alcatel-Lucent Enterprise, France

Abstract

Parameter-efficient fine-tuning is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must acquire anonymized geographic associations while retaining unrelated capabilities. Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition--retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining. Code and data are available at https://github.com/zhngstl/new_facts_forgetting.

Figures & tables

Split	Names	EM	Pref. (%)	$\Delta$ lp
Train facts	Non-anon.	8.10	71.23	3.87
Train facts	Anon.	2.43	58.46	0.26
Paraph. eval	Non-anon.	15.67	68.98	4.02
Paraph. eval	Anon.	9.89	59.33	0.84

Table 1: Base-model prior diagnostic. Anonymization reduces EM accuracy, gold-answer preference, and log-probability margins, suggesting that real names activate relevant pretrained information. The higher paraphrase EM partly reflects its larger share of constrained-response questions; see Appendix A.1 .

Figure 1: OSM paraphrase accuracy against average OOD performance. Points show final-checkpoint means and error bars show standard deviations over five seeds. Higher-rank QLoRA reaches stronger acquisition but lower retention, while FFT and rank 8 remain closer to the pretrained model.

Figure 2: Model-drift diagnostics for different QLoRA ranks. (a) Higher-rank QLoRA adapters show larger KL divergence from the pretrained model, (b) larger effective weight updates, and (c) larger spectral shifts under the SVD intruder diagnostic. Dashed lines show FFT for comparison. Points show means and error bars show std over five seeds.

	Target 75		Target 85		Target 95
Method	Para.	Ret.	Para.	Ret.	Para.	Ret.
FFT	74.9	97.6	76.1	96.9	76.1	96.9
QL r=8	75.7	97.9	79.4	97.5	79.4	97.5
QL r=16	79.3	94.3	85.8	91.1	93.3	89.9
QL r=32	85.2	79.8	85.4	83.0	94.6	73.3
QL r=64	81.0	36.7	83.1	34.3	90.4	38.8

Table 2: Target-acquisition checkpoint comparison. For each target paraphrase accuracy, we select the nearest evaluated checkpoint per seed and method. We report mean achieved paraphrase accuracy and OOD retention as a percentage of base-model OOD performance. The table abbreviates QLoRA as QL.

Method	MATH-500	Minerva	Olympiad Bench	AMC’23	AIME’24	AIME’25	Math Avg.	OOD Avg.	OOD Drop $\downarrow$
FFT	78.80	34.56	44.96	60.00	20.00	16.67	42.50	71.12	1.71
QLoRA $r=16$	77.40	34.19	40.65	60.00	20.00	23.33	42.60	70.61	2.23
QLoRA $r=32$	78.60	37.13	43.92	62.50	13.33	16.67	42.03	71.25	1.58

Table 3: Math-adaptation results. Pass@1 scores and averages are in percent. OOD averages cover HumanEval, IFEval, TruthfulQA, MMLU-Redux, and BBH; OOD drop is relative to the base Qwen3-4B.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Open-ended	4-choice	Yes/no	Random Chance
Train	1,540 (79.5%)	278 (14.3%)	120 (6.2%)	6.68%
Para.	647 (71.9%)	133 (14.8%)	120 (13.3%)	10.36%

Appendix

Table 4: Response-format composition. Counts and within-split ratios for training and paraphrase splits, with approximate chance EM for each split.

Method	HumanEval	IFEval	TruthfulQA	MMLU-Redux	BBH	OOD Avg.
FFT	79.2 $\pm$ 2.2	80.2 $\pm$ 0.4	50.5 $\pm$ 0.6	72.7 $\pm$ 0.2	70.8 $\pm$ 1.2	70.7 $\pm$ 0.7
QLoRA $r=8$	78.4 $\pm$ 1.8	80.6 $\pm$ 1.3	51.6 $\pm$ 1.2	73.0 $\pm$ 0.1	69.9 $\pm$ 2.1	70.7 $\pm$ 0.7
QLoRA $r=16$	70.9 $\pm$ 3.0	70.3 $\pm$ 1.8	50.3 $\pm$ 1.0	69.9 $\pm$ 0.2	57.7 $\pm$ 4.3	63.8 $\pm$ 1.7
QLoRA $r=32$	59.9 $\pm$ 4.0	34.5 $\pm$ 5.8	51.6 $\pm$ 2.0	59.0 $\pm$ 4.3	34.9 $\pm$ 6.4	48.0 $\pm$ 3.6
QLoRA $r=64$	23.2 $\pm$ 16.5	14.6 $\pm$ 2.4	47.7 $\pm$ 1.7	37.6 $\pm$ 11.5	17.3 $\pm$ 4.0	28.1 $\pm$ 6.3

Appendix

Table 5: Per-benchmark OOD scores at the final checkpoint (mean

\pm

standard deviation over five seeds). Degradation is broad on HumanEval, IFEval, MMLU-Redux, and BBH; TruthfulQA is comparatively stable.

Hyperparameter	QLoRA	Full fine-tuning
Base model	Qwen3-4B
Training samples	1,938
Epochs	100
Batch size	16
Learning rate	$2\times 10^{-4}$	$2\times 10^{-5}$
LR scheduler	Linear

Appendix

Table 6: Hyperparameters for the main OpenStreetMap fine-tuning experiments.

Hyperparameter	QLoRA	Full fine-tuning
Base model	Qwen3-4B
Dataset	open-r1-math-220k
Training samples	94k
Epochs	1
Batch size	$2\times 16$
Learning rate	$1\times 10^{-5}$	$2\times 10^{-5}$

Appendix

Table 7: Hyperparameters for the additional math adaptation experiments.

Factual Knowledge in Language Models

Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.

See all 195 papers

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
May 16, 2026Ramakrishnan SathyavageeswaranAdapter Tuning Low-Rank Adaptation
Intuit
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China

May 8, 2026J/K move · Enter open · S save

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li+2

Carnegie Mellon University · Tsinghua University · Infinigence AI

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater representational plasticity, Low-Rank Adaptation (LoRA) can match or surpass FFT performance while constraining updates to a low-rank space and potentially benefiting from additional regularization. Through empirical evaluation across diverse tasks (SQL, Medical QA, and Counterfactual Knowledge) and varying language models (Gemma-3-1B, Qwen2.5-1.5B, and Qwen2.5-3B), we observe both trends and find that the better static architecture depends on the task and model. Spectral and truncation analyses further show that endpoint compressibility alone does not explain these task differences, suggesting task-score sensitivity and constrained optimization trajectories as possible explanations. To address this challenge, we propose a Mixture of LoRA and Full (MoLF) Fine-Tuning, a unified framework that enables continuous navigation between both training regimes. MoLF dynamically routes updates between FFT and LoRA at the optimizer level to ensure that exact gradient signals are available to both experts throughout training, while only selected experts update their weights. For memory-constrained environments, we also introduce MoLF-Efficient, which freezes base weights and only routes updates among a pair of LoRA experts of potentially varying rank. Our evaluations show that MoLF either improves on or stays within

1.5

percentage points of the better of FFT and LoRA across the nine tested settings, while MoLF-Efficient outperforms both AdaLoRA and AdaMix in eight of nine settings, with gains over the stronger baseline of up to

11.70

percentage points on Fact,

3.13

on Med, and

2.98

on SQL.

Abstract

Factual Knowledge in Language Models

Explore similar work

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning