MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning

Sten Rüdiger; Sebastian Raschka

cs.LGApr 2, 2026

MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning

Authors: Sten Rüdiger, Sebastian Raschka

Organizations: Independent Researcher · RAIR Lab

Abstract

Minor Component Adaptation (MiCA) is a novel parameter-efficient fine-tuning method for large language models that focuses on adapting underutilized subspaces of model representations. Unlike conventional methods such as Low-Rank Adaptation (LoRA), which target dominant subspaces, MiCA leverages Singular Value Decomposition to identify subspaces related to minor singular vectors associated with the least significant singular values and constrains the update of parameters during fine-tuning to those directions. This strategy leads to up to 5.9x improvement in knowledge acquisition under optimized training hyperparameters and a minimal parameter footprint of 6-60% compared to LoRA. These results suggest that constraining adaptation to minor singular directions provides a more efficient and stable mechanism for integrating new knowledge into pre-trained language models.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Diagram of a LoRA-style low-rank adaptation module applied to a weight matrix. (b) Diagram of a MiCA-style adaptation module illustrating its distinct update structure compared to LoRA (blue: constrained components, orange: fine-tuned components initialized as indicated in each trapezoid). $U[:,-r:]$ means all rows, last $r$ columns of the matrix $U$ defined in the text.

Panel of Figure 2 — Figure 2: Retention of domain knowledge from the BLOGS dataset as a function of number of training epochs for MiCA and LoRA. Error bars denote the standard error of 8 evaluation runs. The horizontal dashed line shows the baseline performance of the respective foundation model. The green dashed line in (b) shows results for a fixed $B$ matrix initialized with random SVD components instead of the minor componenents and same hyperparameters as for the MiCA runs.

Method	BLOGS-MC	TruthQA	HellaSwag	r	LR	Ep.	Params
Llama-2-7B-chat	56.18	34.79	60.40	–	–	–	6,747M
LoRA (optimal)	58.28	35.47	60.41	128	1e-4	8	67M
MiCA (optimal)	61.33	35.29	60.11	16	5e-4	4	4M
Qwen2.5-7B-Instr.	72.91	43.27	60.60	–	–	–	7,626M
LoRA (optimal)	73.87	42.95	60.95	32	5e-4	4	10M
MiCA (optimal)	75.63	43.38	61.62	32	5e-4	8	6M

Table 1: Comparison of BLOGS-MC (new knowledge), TruthfulQA-MC1 and HellaSwag accuracies across different fine-tuning methods for the blog post training. For each accuracy value we averaged 8 runs per hyperparameter choice. Additional columns for optimal adapter rank (

r

), learning rate (LR), and training epochs are included for completeness. The last columns shows the number of parameters of the full model or the number of trainable parameters for the LoRA and MiCA runs, respectively.

Method	Accuracy (%)	HellaSwag (%)	Learning rate	r	# epochs
Llama-2-7B-chat	27.4	57.8	–	–	–
Full FT (chat)	30.4	57.3	2e-5	–	1
LoRA	29.4 1 1 footnotemark: 1	57.7	5e-4	32	8
MiCA	39.2 2 2 footnotemark: 2	57.8	2e-3	32	8

Table 2: Retention of new knowledge for the HISTORY dataset: accuracy on multiple choice tests for several fine-tuning methods (Llama-2) with respective optimal hyperparameter choices. ( 1 mean of results for two runs, 2 identical results across four independent runs)

Figure 3: Retention of domain knowledge for the history book and for several fine-tuning methods for Llama-2.

Method	BLOGS-MC (%)
Instruct (no FT)	72.91
Major- $r$	74.21
$r$ random components	73.75
Minor- $r$ (MiCA)	75.63

Table 3: Ablation on Qwen-2.5-7B comparing adaptation in major vs. minor and randomly chosen singular subspaces.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Per-device train batch size	4
Gradient accumulation steps	1
Weight decay	0.01
Warmup ratio	0.1
LR scheduler	cosine
Max grad norm	0.3

Appendix

Table 4: SFT hyperparameters used in all runs (TRL SFTConfig ).

Parameter	Value
Dropout	0.05
Target modules	q_proj , v_proj
Per-device train batch size	auto
Gradient accumulation steps	1
Weight decay	0.01
Warmup ratio	0.1

Appendix

Table 5: LoRA and MiCA adapter configurations and trainings hyperparameters.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 755 papers

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching
Oct 26, 2026Fei Zuo, Yizhou Huang, Kezhi WangLow-Rank Adaptation Parameter-Efficient Fine-Tuning
East China Normal University · Brunel University of London

May 8, 2026J/K move · Enter open · S save

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li+2

Carnegie Mellon University · Tsinghua University · Infinigence AI

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater representational plasticity, Low-Rank Adaptation (LoRA) can match or surpass FFT performance while constraining updates to a low-rank space and potentially benefiting from additional regularization. Through empirical evaluation across diverse tasks (SQL, Medical QA, and Counterfactual Knowledge) and varying language models (Gemma-3-1B, Qwen2.5-1.5B, and Qwen2.5-3B), we observe both trends and find that the better static architecture depends on the task and model. Spectral and truncation analyses further show that endpoint compressibility alone does not explain these task differences, suggesting task-score sensitivity and constrained optimization trajectories as possible explanations. To address this challenge, we propose a Mixture of LoRA and Full (MoLF) Fine-Tuning, a unified framework that enables continuous navigation between both training regimes. MoLF dynamically routes updates between FFT and LoRA at the optimizer level to ensure that exact gradient signals are available to both experts throughout training, while only selected experts update their weights. For memory-constrained environments, we also introduce MoLF-Efficient, which freezes base weights and only routes updates among a pair of LoRA experts of potentially varying rank. Our evaluations show that MoLF either improves on or stays within

1.5

percentage points of the better of FFT and LoRA across the nine tested settings, while MoLF-Efficient outperforms both AdaLoRA and AdaMix in eight of nine settings, with gains over the stronger baseline of up to

11.70

percentage points on Fact,

3.13

on Med, and

2.98

on SQL.

Abstract

LLM Fine-Tuning

Explore similar work

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching