Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

Abdulmalek Saket

cs.LGApr 4, 2026

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

Authors: Abdulmalek Saket

Organizations: Royal Fenice Kft / ALETHEIA PROTOCOL research, Budapest, Hungary

Abstract

Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large language models, yet standard practice applies LoRA adapters uniformly to all transformer layers regardless of their relevance to the downstream task. We introduce Aletheia, a gradient-guided layer selection method that identifies the most task-relevant layers via a lightweight gradient probe and applies LoRA adapters only to those layers with asymmetric rank allocation. Across 81 experiment rows covering 14 successful models from 8 architecture families (0.5B-72B parameters, including dense and Mixture-of-Experts architectures), with one additional documented failed Pythia/GPT-NeoX attempt in Campaign 2, Aletheia achieves a 15-28% training speedup (mean 23.1%, p < 0.001) with bounded extra forgetting and broadly matched downstream behavior on the evaluated MMLU, GSM8K, and HumanEval benchmark pack. Across the tested families and scales, Campaign 1 shows a 100% per-model speed win rate and Campaign 2 shows broadly preserved downstream behavior within a bounded-degradation framing. Together these results support a practical model-economics claim: intelligent layer selection can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.

Figures & tables

Model	Family	Params	Layers	Campaign
Qwen2.5-0.5B	Qwen	0.5B	24	1
TinyLlama-1.1B	TinyLlama	1.1B	22	1
Qwen2.5-1.5B	Qwen	1.5B	28	1
StableLM-2-1.6B	StableLM	1.6B	24	2
Qwen2.5-3B	Qwen	3B	36	2
Phi-3.5-mini	Phi	3.8B	32	1

Table 1: Models evaluated across two experimental campaigns.

Model	Std Time (s)	Ale Time (s)	Speedup (%)	Ratio	$p$ -value
Qwen2.5-0.5B	$93.9{\pm}1.5$	$69.4{\pm}1.0$	$26.1{\pm}0.5$	1.353 $\times$	0.0006
TinyLlama-1.1B	$90.6{\pm}1.0$	$66.7{\pm}0.6$	$26.3{\pm}0.2$	1.357 $\times$	0.0002
Qwen2.5-1.5B	$113.0{\pm}0.1$	$82.4{\pm}0.1$	$27.1{\pm}0.1$	1.371 $\times$	0.0000
Phi-3.5-mini	$105.5{\pm}1.1$	$86.2{\pm}1.2$	$18.4{\pm}0.4$	1.225 $\times$	0.0003
Mistral-7B	$133.1{\pm}1.4$	$96.2{\pm}0.8$	$27.7{\pm}0.2$	1.383 $\times$	0.0002
Qwen2.5-7B	$157.5{\pm}0.6$	$129.7{\pm}0.5$	$17.6{\pm}0.1$	1.214 $\times$	0.0000

Table 2: Training speedup of Aletheia vs. Standard LoRA (Campaign 1, 3-seed mean

\pm

SD).

Figure 1: Training speedup of Aletheia vs. Standard LoRA across 10 models (3-seed means with 95% CI error bars). All models show positive speedup with tight confidence intervals, confirming reproducibility.

Model	Base	Std FT	Std $\Delta$	Ale FT	Ale $\Delta$	Extra
Qwen2.5-0.5B	37.3%	36.2%	$-$ 1.2pp	36.7%	$-$ 0.7pp	+0.5pp
TinyLlama-1.1B	23.5%	21.7%	$-$ 1.8pp	23.5%	+0.0pp	+1.8pp
Qwen2.5-1.5B	46.0%	45.2%	$-$ 0.8pp	44.2%	$-$ 1.8pp	$-$ 1.0pp
Phi-3.5-mini	23.0%	25.5%	+2.5pp	26.2%	+3.2pp	+0.7pp
Mistral-7B	40.3%	39.8%	$-$ 0.5pp	41.3%	+1.0pp	+1.5pp
Qwen2.5-7B	49.0%	50.3%	+1.3pp	48.2%	$-$ 0.8pp	$-$ 2.2pp

Table 3: MMLU forgetting analysis (Campaign 1, 3-seed means). Forgetting

\leq 2

pp for all models.

Model	$\Delta$ MMLU (pp)	$\Delta$ GSM8K (pp)	$\Delta$ HumanEval (pp)
Qwen2.5-3B	$-$ 0.60	$-$ 0.67	$+$ 1.63
Qwen2.5-7B	$-$ 0.54	$-$ 1.50	$-$ 0.61
Llama-3.1-8B	$+$ 0.01	$+$ 0.00	$+$ 1.53
StableLM-2-1.6B	$-$ 1.33	$+$ 3.50	$+$ 1.42
GPT-J-6B	$+$ 0.50	$+$ 2.17	$+$ 0.00
Mixtral-8 $\times$ 7B	$-$ 0.17	$-$ 1.67	$+$ 0.00

Table 4: Downstream benchmark deltas (Aletheia

-

Standard, 3-seed means, Campaign 2). Values near zero indicate matched performance.

Figure 2: Campaign 2 benchmark deltas (Standard LoRA vs. Aletheia) with 95% CI error bars across 6 models and 3 benchmarks. Taken together with the per-model means, the intervals support a bounded-delta interpretation on the evaluated set rather than a quality-collapse story.

Figure 3: Speedup by architecture family (Campaign 1, 95% CI). All 5 Campaign 1 families show consistent, statistically significant speedup from gradient-guided layer selection.

Model	Eval Loss (Std / CM-Ale)	Downstream Match?
Qwen2.5-3B	0.311 / 0.368	Mixed / roughly neutral
Qwen2.5-7B	0.309 / 0.354	Functionally equivalent
Llama-3.1-8B	0.648 / 0.716	Partial downstream evidence

Table 5: Compute-matched Aletheia vs. Standard LoRA (Campaign 2, 3-seed means).

Figure 4: Benchmark deltas (percentage points) for Standard LoRA, Aletheia (matched steps), and Aletheia (compute-matched) across three core models. The available downstream evaluations stay in a bounded-delta regime, supporting the claim that Aletheia’s speed savings do not induce a clear material downstream penalty on the evaluated set.

Figure 5: Speed–quality trade-off across Campaign 1 models. Each point represents a model’s mean speedup (

x

) vs. extra MMLU forgetting (

y

) with 95% CI error bars. Most models cluster near the zero-forgetting band, supporting a bounded-degradation interpretation rather than a quality-collapse tradeoff.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching
Oct 26, 2026Fei Zuo, Yizhou Huang, Kezhi WangLow-Rank Adaptation Parameter-Efficient Fine-Tuning
East China Normal University · Brunel University of London
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2LLM Fine-Tuning Efficient Language Model Training
University of Wisconsin-Madison · Amazon Web Services

Oct 26, 2026J/K move · Enter open · S save

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Fei Zuo, Yizhou Huang, Kezhi Wang

East China Normal University · Brunel University of London

Low-Rank Adaptation Parameter-Efficient Fine-Tuning Mechanistic Interpretability

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA applies adaptation throughout the transformer, despite heterogeneous contributions from its components. In this work, we conduct a systematic mechanistic interpretability analysis to understand how LoRA affects model internals. We observe that LoRA's adaptation effect is highly non-uniform, with significant component-wise and layer-wise sparsity. Most critically, we find that important layers are highly task-dependent, with limited overlap across tasks. Motivated by these insights, we propose Adaptive-LoRA, a calibration-driven method that automatically identifies task-specific layers via frozen-base zero-ablation, enabling efficient layer selection without exhaustive search. Extensive experiments on Qwen2.5-1.5B across 11 diverse reasoning tasks demonstrate that Adaptive-LoRA outperforms full-layer LoRA while using only 30% of layers, achieving a

3.5\times

reduction in adapter parameters. This work bridges mechanistic interpretability and practical efficiency, providing both fundamental insights and a principled approach for adaptive fine-tuning.

Abstract

LLM Fine-Tuning

Explore similar work

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

PLoRA: Efficient Concurrent LoRA Training for Large Language Models