HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Bartosz Trojan; Filip Gębala

cs.CLMar 1, 2026

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Authors: Bartosz Trojan, Filip Gębala

Organizations: Upper-Secondary Schools of Communications in Cracow

Abstract

Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA

Figures & tables

Figure 1: A hyper-network generates the weights for the Query and Value matrices in all attention blocks of the RoBERTa model, while the original pretrained weights remain frozen. The figure illustrates the approach in which the hyper-network produces both the $A$ and $B$ matrices. In this work, we also present a variant where only the $B$ matrices are generated by the hyper-network, and $A$ matrices are fixed with randomly initialized values, which isn’t shown on this figure.

	Metric	CoLA	SST-2	QNLI	MRPC	RTE	MNLI
FT	$Score$ $\uparrow$	61.68 $\pm_{0.73}$	94.50 $\pm_{0.41}$	92.64 $\pm_{0.22}$	90.37 $\pm_{0.41}$	85.32 $\pm_{1.37}$	87.17
	ECE $\downarrow$	0.136 $\pm_{0.008}$	0.035 $\pm_{0.017}$	0.072 $\pm_{0.002}$	0.111 $\pm_{0.023}$	0.104 $\pm_{0.045}$	0.074
	CECE $\downarrow$	0.138 $\pm_{0.007}$	0.036 $\pm_{0.016}$	0.072 $\pm_{0.002}$	0.114 $\pm_{0.021}$	0.113 $\pm_{0.038}$	0.050
	MCE $\downarrow$	0.339 $\pm_{0.074}$	0.277 $\pm_{0.242}$	0.539 $\pm_{0.111}$	0.308 $\pm_{0.163}$	0.309 $\pm_{0.159}$	0.202
	ACE $\downarrow$	0.134 $\pm_{0.009}$	0.033 $\pm_{0.016}$	0.071 $\pm_{0.002}$	0.110 $\pm_{0.021}$	0.103 $\pm_{0.039}$	0.073
	TACE 0.01 $\downarrow$	0.469 $\pm_{0.006}$	0.461 $\pm_{0.013}$	0.482 $\pm_{0.007}$	0.441 $\pm_{0.020}$	0.413 $\pm_{0.058}$	0.492

Table 1: Performance and calibration metrics across GLUE benchmarks for RoBERTa

{}_{\text{large}}

. FT denotes standard fine-tuning [ 16 ] ; LoRA denotes Low-Rank Adaptation [ 10 ] . Following [ 24 ] , we report Matthews Correlation Coefficient (MCC) for CoLA, F1 for MRPC, and Accuracy for other tasks. Minor score deviations from [ 10 ] are attributable to different random seeds; additionally, MRPC reports F1 (as in [ 16 ] ) rather than accuracy, and RTE (FT) is initialized from MNLI-pretrained weights for fair comparison [ 10 ] .

	COLA		SST-2
	MCC $\uparrow$	ECE $\downarrow$	Acc $\uparrow$	ECE $\downarrow$
MLP $A_{\text{gen}}$	$63.54_{\pm 0.31}$	$0.116_{\pm 0.019}$	$94.84_{\pm 0.30}$	$0.037_{\pm 0.007}$
MLP $A_{\text{fix}}$	$48.25_{\pm 13.01}$	$0.130_{\pm 0.015}$	$92.89_{\pm 1.41}$	$0.047_{\pm 0.011}$
Transformer $A_{\text{gen}}$	$\mathbf{64.42_{\pm 1.75}}$	$0.119_{\pm 0.009}$	$94.78_{\pm 0.08}$	$0.040_{\pm 0.003}$
Transformer $A_{\text{fix}}$	$60.69_{\pm 0.35}$	$\mathbf{0.100_{\pm 0.010}}$	$94.56_{\pm 0.08}$	$\mathbf{0.028_{\pm 0.004}}$
LoRA	$63.94_{\pm 0.21}$	$0.120_{\pm 0.025}$	$\mathbf{94.99_{\pm 0.18}}$	$0.046_{\pm 0.001}$

Table 2: We compare four hypernetwork configurations combining two architectures (MLP and Transformer) with two weight generation strategies (

A_{\text{gen}}

and

A_{\text{fix}}

), each averaged over 3 seeds. Transformer

A_{\text{gen}}

outperforms the LoRA baseline on CoLA, though LoRA remains stronger on SST-2, suggesting task-dependent behavior. Interestingly, Transformer

A_{\text{fix}}

achieves the best calibration across all runs despite not leading in Matthews Correlation Coefficient (MCC) or Accuracy.

Figure 2: Evaluation results on CoLA and SST-2 benchmarks, reported as Matthews Correlation Coefficient and accuracy (top row) alongside Expected Calibration Error (bottom row), averaged across 3 independent random seeds.

A_{\text{gen}}

means both matrices

A

and

B

are generated, and

A_{\text{fix}}

means matrices

A

are fixed. LoRA [ 10 ] is included as a baseline. Fixing matrix

A

improves model calibration, albeit at the cost of task performance across both datasets.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 538 papers

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
Jan 28, 2026Moule Lin, Shuhao Guan, Andrea Patane +2Bayesian Neural Networks Confidence Estimation in Language Models
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland
EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning
Aug 2, 2026Srinivas Anumasa, Dianbo LiuUncertainty Quantification Low-Rank Adaptation
National University of Singapore
LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
Aug 4, 2026Linhan Xia, Rui Liu, Zhaofeng Zhang +3Fine-Tuning LLM Fine-Tuning
University of Oklahoma · 4Tencent · 2Imperial College London +5

Jan 28, 2026J/K move · Enter open · S save

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Moule Lin, Shuhao Guan, Andrea Patane, David Gregg+1

School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Bayesian Neural Networks Confidence Estimation in Language Models LLM Fine-Tuning Low-Rank Adaptation Language Model Calibration LLM Reliability+2

Open paper AskPDF arXiv

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and

{\approx}1.2{\times}

training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Abstract

Low-Rank Adaptation

Explore similar work

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment