cs.CLMar 1, 2026

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Authors: Bartosz Trojan, Filip Gębala

Organizations: Upper-Secondary Schools of Communications in Cracow

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA

Figures & tables

Figure 1: A hyper-network generates the weights for the Query and Value matrices in all attention blocks of the RoBERTa model, while the original pretrained weights remain frozen. The figure illustrates the approach in which the hyper-network produces both the AA and BB matrices. In this work, we also present a variant where only the BB matrices are generated by the hyper-network, and AA matrices are fixed with randomly initialized values, which isn’t shown on this figure.
 MetricCoLASST-2QNLIMRPCRTEMNLI
FTScoreScore ↑\uparrow61.68 ±0.73\pm_{0.73}94.50 ±0.41\pm_{0.41}92.64 ±0.22\pm_{0.22}90.37 ±0.41\pm_{0.41}85.32 ±1.37\pm_{1.37}87.17
ECE ↓\downarrow0.136 ±0.008\pm_{0.008}0.035 ±0.017\pm_{0.017}0.072 ±0.002\pm_{0.002}0.111 ±0.023\pm_{0.023}0.104 ±0.045\pm_{0.045}0.074
CECE ↓\downarrow0.138 ±0.007\pm_{0.007}0.036 ±0.016\pm_{0.016}0.072 ±0.002\pm_{0.002}0.114 ±0.021\pm_{0.021}0.113 ±0.038\pm_{0.038}0.050
MCE ↓\downarrow0.339 ±0.074\pm_{0.074}0.277 ±0.242\pm_{0.242}0.539 ±0.111\pm_{0.111}0.308 ±0.163\pm_{0.163}0.309 ±0.159\pm_{0.159}0.202
ACE ↓\downarrow0.134 ±0.009\pm_{0.009}0.033 ±0.016\pm_{0.016}0.071 ±0.002\pm_{0.002}0.110 ±0.021\pm_{0.021}0.103 ±0.039\pm_{0.039}0.073
TACE 0.01 ↓\downarrow0.469 ±0.006\pm_{0.006}0.461 ±0.013\pm_{0.013}0.482 ±0.007\pm_{0.007}0.441 ±0.020\pm_{0.020}0.413 ±0.058\pm_{0.058}0.492
Table 1: Performance and calibration metrics across GLUE benchmarks for RoBERTa large{}_{\text{large}} . FT denotes standard fine-tuning [ 16 ] ; LoRA denotes Low-Rank Adaptation [ 10 ] . Following [ 24 ] , we report Matthews Correlation Coefficient (MCC) for CoLA, F1 for MRPC, and Accuracy for other tasks. Minor score deviations from [ 10 ] are attributable to different random seeds; additionally, MRPC reports F1 (as in [ 16 ] ) rather than accuracy, and RTE (FT) is initialized from MNLI-pretrained weights for fair comparison [ 10 ] .
 COLASST-2
 MCC ↑\uparrowECE ↓\downarrowAcc ↑\uparrowECE ↓\downarrow
MLP AgenA_{\text{gen}}63.54±0.3163.54_{\pm 0.31}0.116±0.0190.116_{\pm 0.019}94.84±0.3094.84_{\pm 0.30}0.037±0.0070.037_{\pm 0.007}
MLP AfixA_{\text{fix}}48.25±13.0148.25_{\pm 13.01}0.130±0.0150.130_{\pm 0.015}92.89±1.4192.89_{\pm 1.41}0.047±0.0110.047_{\pm 0.011}
Transformer AgenA_{\text{gen}}64.42±1.75\mathbf{64.42_{\pm 1.75}}0.119±0.0090.119_{\pm 0.009}94.78±0.0894.78_{\pm 0.08}0.040±0.0030.040_{\pm 0.003}
Transformer AfixA_{\text{fix}}60.69±0.3560.69_{\pm 0.35}0.100±0.010\mathbf{0.100_{\pm 0.010}}94.56±0.0894.56_{\pm 0.08}0.028±0.004\mathbf{0.028_{\pm 0.004}}
LoRA63.94±0.2163.94_{\pm 0.21}0.120±0.0250.120_{\pm 0.025}94.99±0.18\mathbf{94.99_{\pm 0.18}}0.046±0.0010.046_{\pm 0.001}
Table 2: We compare four hypernetwork configurations combining two architectures (MLP and Transformer) with two weight generation strategies ( AgenA_{\text{gen}} and AfixA_{\text{fix}} ), each averaged over 3 seeds. Transformer AgenA_{\text{gen}} outperforms the LoRA baseline on CoLA, though LoRA remains stronger on SST-2, suggesting task-dependent behavior. Interestingly, Transformer AfixA_{\text{fix}} achieves the best calibration across all runs despite not leading in Matthews Correlation Coefficient (MCC) or Accuracy.
Figure 2: Evaluation results on CoLA and SST-2 benchmarks, reported as Matthews Correlation Coefficient and accuracy (top row) alongside Expected Calibration Error (bottom row), averaged across 3 independent random seeds. AgenA_{\text{gen}} means both matrices AA and BB are generated, and AfixA_{\text{fix}} means matrices AA are fixed. LoRA [ 10 ] is included as a baseline. Fixing matrix AA improves model calibration, albeit at the cost of task performance across both datasets.
More in this topic

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 538 papers

Explore similar work

CardsList
Jan 28, 2026

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ≈1.2×{\approx}1.2{\times} training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Moule Lin, Shuhao Guan, Andrea Patane +2

School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland and Lero the Research Ireland Centre for Software, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

Bayesian Neural NetworksConfidence Estimation in Language ModelsLLM Fine-TuningLow-Rank AdaptationLanguage Model CalibrationLLM Reliability+2
AskPDF
Aug 2, 2026

EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation. We introduce EulerLoRA, a stochastic extension of LoRA that generates multiple predictive trajectories by sampling structured variations along the rank-one components of shared low-rank adapters, while preserving the deterministic LoRA transformation in expectation. We evaluate EulerLoRA with vision transformers on CIFAR-10, CIFAR-100, and HAM10000, together with out-of-distribution detection on SVHN. Across these benchmarks, EulerLoRA achieves comparable or improved performance relative to strong LoRA-Ensemble baselines. Using two rank-20 adapters, EulerLoRA requires approximately 3 million trainable adapter parameters, compared with about 10 million for a rank-8, 16-adapter LoRA-Ensemble, corresponding to roughly 69% fewer trainable parameters. These results show that useful predictive diversity can be obtained from a small number of shared adapters.

Srinivas Anumasa, Dianbo Liu

National University of Singapore

Uncertainty QuantificationLow-Rank AdaptationParameter-Efficient Fine-Tuning
AskPDF
Aug 4, 2026

LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone. Every adaptation step therefore needs backward-capable hardware and must store or recompute activations. We ask whether this repeated backward chain can be replaced by a one-time calibration. We introduce Local Credit Assignment (LoCA), a two-stage method for small-shift adaptation. One probe backward pass fits a low-rank map at each transformer block from the final prediction error to a local hidden-state correction. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low-rank adapters with closed-form ridge solves. No further backbone backward pass is required. We evaluate LoCA on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B. In 16 of 25 reported task--scale comparisons, LoCA yields lower evaluation cross-entropy than the corresponding LoRA run. Its measured full-run GPU peak, including calibration, is 26--29% lower than LoRA's. After calibration, its CPU steady-state memory is 36--52% lower and its per-pass time is 43--48% lower. A shared scale-normalized candidate set is reused across all tested Qwen2.5 sizes and on SmolLM2-1.7B. LoCA thus amortizes global credit assignment into one calibration and enables later forward-only tuning when repeated backpropagation is impractical. The code associated with this paper is available here.

Linhan Xia, Rui Liu, Zhaofeng Zhang +3

University of Oklahoma · 4Tencent · 2Imperial College London +5

Fine-TuningLLM Fine-TuningLow-Rank AdaptationCredit AssignmentParameter-Efficient Fine-Tuning
AskPDF