Adapter Tuning

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 56

All topics

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
May 16, 2026Ramakrishnan SathyavageeswaranAdapter Tuning Low-Rank Adaptation
Intuit
PreFT: Prefill-only finetuning for efficient inference
May 14, 2026Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4Fine-Tuning LLM Serving
Stanford University · Tilde Research
MinT: Managed Infrastructure for Training and Serving Millions of LLMs
May 13, 2026Mind Lab, :, Song Cao +60LLM Serving Adapter Tuning
Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
May 12, 2026Arijit Sehanobish, Charles LoveringAdapter Tuning Low-Rank Adaptation
Kensho Technologies
TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
May 12, 2026Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang +3Audio-Visual Understanding Audio-Visual Representation Learning
AI2 Lab, KAIST
Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary
May 11, 2026Ramchand KumaresanAdaptive Model Routing Evolutionary Optimization
Murai Labs
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
May 11, 2026Barbara Su, Fangshuo Liao, Anastasios KyrillidisLLM Fine-Tuning Adapter Tuning
Department of Computer Science Rice University
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
May 9, 2026Luca BalloreLanguage Model Pretraining Adapter Tuning
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
May 8, 2026Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho +1Memory-Augmented Neural Networks Adapter Tuning
Department of Computer Science University of Texas at Austin · Department of Government University of Texas at Austin · University of Texas at Austin
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
May 8, 2026Ionut-Vlad Modoranu, Mher Safaryan, Dan AlistarhLLM Fine-Tuning Hierarchical Representation Learning
Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA
DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
May 8, 2026Mengxin Qin, Xiang Zhang, Xi Wang +3Vision-Language Models Fine-Tuning
School of Electronic Engineering, Xidian University Xi’an 710071, China
Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
May 7, 2026Suoxin Zhang, Run He, Di Fang +3LLM Fine-Tuning Adapter Tuning
South China University of Technology, China · Zhejiang University, China
TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models
May 7, 2026Duong Nguyen, Mohammed Jawhar, Nicolas ChesneauTabular Foundation Models Adapter Tuning
Ekimetrics
EGA: Adapting Frozen Encoders for Vector Search with Bounded Out-of-Distribution Degradation
May 7, 2026Dongfang ZhaoOOD Generalization Vision Foundation Model Adaptation
Tacoma School of Engineering and Technology University of Washington
Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces
May 4, 2026Jingze Ge, Yun Liu, Xue Geng +4Model Compression Adapter Tuning
National University of Singapore · Nankai University · Institute for Infocomm Research (I2R), A*STAR
SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
May 2, 2026Shuaipeng Zhou, Yu ZhangUncertainty Quantification Adapter Tuning
School of Computer Science and Technology, Shenyang University of Chemical Technology · 2Liaoning Key Laboratory of Intelligent Technology for Chemical Process Industry
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
May 2, 2026Zhiwen Ruan, Yichao Du, Jianjie Zheng +6LLM Fine-Tuning Large Language Model-Guided Optimization
Southern University of Science and Technology · Alibaba Group · Shanghai University of Finance and Economics +2
BoostLoRA: Growing Effective Rank by Boosting Adapters
Apr 30, 2026Raviteja Anantha, Nick Levato, Layne C. PriceLLM Fine-Tuning Adapter Tuning
Amazon
Low-Rank Adaptation Redux for Large Models
Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisAdapter Tuning Low-Rank Adaptation
Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA
Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
Apr 18, 2026Yixuan Tang, Yi YangAdapter Tuning Low-Rank Adaptation
The Hong Kong University of Science and Technology
Introspection Adapters: Training LLMs to Report Their Learned Behaviors
Apr 18, 2026Keshav Shenoy, Li Yang, Abhay Sheshadri +4LLM Auditing Language Model Introspection
Anthropic Fellows · Ashwood Centre on AI Science and Policy, University of Cambridge · Anthropic
SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation
Mar 14, 2026Anbang Wang, Yuzhuo Ao, Shangzhe Wu +13D Foundation Models 3D Asset Generation
The Hong Kong University of Science and Technology · University of Cambridge
CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference
Feb 26, 2026Hung-Hsuan ChenAdapter Tuning Low-Rank Adaptation
Computer Science and Information Engineering National Central University Taoyuan, Taiwan
MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning
Jun 10, 2025Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray +3Transformer Tensor Decomposition
Global Technology Applied Research, JPMorgan Chase, New York, NY 10001, USA
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Date pendingXianming Li, Zongxi Li, Tsz-fung Andrew Lee +3Fine-Tuning Adapter Tuning
PolyU · School of Data Science, Lingnan University, Hong Kong SAR

May 16, 2026, cs.LGJ/K move · Enter open · S save

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

Ramakrishnan Sathyavageeswaran

Intuit

Adapter Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.

Momentum

Related topics

Latest papers 56