Fine-Tuning

Follow this topic Ask about this topic

Latest papers 205

31 papers in the last four weeks, up 72% on the four weeks before. 0.2% of all new papers.

All topics

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model
Jun 4, 2026O. Duranthon, F. Boncoraglio, L. ZdeborováFine-Tuning Low-Rank Adaptation
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Jun 3, 2026Wanhao Yu, Ziyan Wang, Zheng Wang +7Fine-Tuning Memory-Efficient Fine-Tuning
University of North Carolina at Charlotte · University of Houston · State University of New York at Binghamton +2
Towards Pretraining Text Encoders for TabPFN
Jun 3, 2026Mustafa Tajjar, Alexander Pfefferle, Lennart Purucker +1Tabular Foundation Models Fine-Tuning
Department of Computer Science, University of Freiburg, Freiburg, Germany · ELLIS Institute Tübingen, Tübingen, Germany · Prior Labs, Freiburg, Germany.
(Mis)generalization of Helpful-only Fine-tuning
Jun 3, 2026Mohammad Omar Khursheed, Baram Sosis, Fabien RogerSupervised Fine-Tuning LLM Sycophancy
Anthropic Fellows Program MATS · Anthropic
AutoTail-BSFGM: Class-Balance-Aware Fine-Tuning for Chinese Scholarly Text Classification
Jun 2, 2026Anling Xiang, Yuwen Yang, Yang ShenClass-Imbalanced Learning Long-Tailed Classification
Department of Intelligent Communication, School of Journalism and Communication, Minzu University of China, Beijing, China · ZeeLin (Beijing) Technology Co., Ltd., Beijing, China · School of Journalism and Communication, Tsinghua University, Beijing, China +1
DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data
Jun 2, 2026Yunsheng Yuan, Shaowei Li, Kai Wang +5Fine-Tuning LLM Fine-Tuning
School of Computer Science and Technology, Shandong University, Qingdao China · School of Mathematical Science, Peking University, China · IEIT SYSTEM, China +2
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Fine-Tuning Memory-Efficient Fine-Tuning
University of California, Santa Barbara
Quantifying and Mitigating Domain Shift in Peach Leaf Damage Classification: Attention Mechanisms and Fine-Tuning Strategies
Jun 1, 2026Adrián Cánovas-Rodriguez, Miguel A. González-Illán, Maria Fernanda García-Cruz +5Agricultural Image Analysis Fine-Tuning
Department of Information and Communication Engineering, University of Murcia, Murcia, Spain · Department of Irrigation, Centro de Edafología y Biología Aplicada del Segura CEBAS-CSIC, Murcia, Spain
Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks
Jun 1, 2026Nermeen Abou Baker, David Rohrschneider, Uwe HandmannFine-Tuning Instance Segmentation
Ruhr West University of applied sciences · Lützowstraße 5, 46236 Bottrop, Germany: Ruhr West University of Applied Sciences
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Fine-Tuning Multimodal Large Language Models
University of Virginia
CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs
May 29, 2026Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu +1Continual Learning for LLMs Fine-Tuning
ModelBrew AI
Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)
May 29, 2026Felix AkeretASR Evaluation Fine-Tuning
Independent Researcher, Zurich, Switzerland · Dipl. Ing. ETH Zürich, Dipl. NDS BWI ETH Zürich. · Dipl. NDS BWI ETH Zürich.
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
May 29, 2026Yuduo Li, Xiaofeng Shi, Qian Kou +2Supervised Fine-Tuning Fine-Tuning
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Efficient Test-Time Finetuning of LLMs via Convex Reconstruction and Gradient Caching
May 28, 2026Alaa Khamis, Alaa MaaloufLLM Inference Efficiency Fine-Tuning
Department of Computer Science University of Haifa
A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers
May 28, 2026Hannah Gao, Isha Agarwal, Dylan Hadfield-Menell +1Adversarial Training Fine-Tuning
Massachusetts Institute of Technology · MIT Cambridge, MA
Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models
May 28, 2026Qi Liu, Mingdi Sun, Yongyi He +5Supervised Fine-Tuning Fine-Tuning
University of Science and Technology of China · Huawei Cloud
Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces
May 28, 2026Chen He, Yuhao Wu, Lei Wang +2Fine-Tuning LLM Fine-Tuning
University of Electronic Science and Technology of China · Singapore University of Technology and Design · Singapore Management University
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
May 27, 2026Yangyi Huang, Ruotian Peng, Zeju Qiu +4LLM Evaluation Fine-Tuning
The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
May 26, 2026Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3LLM Inference Efficiency Expert Offloading
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Huawei Technologies Ltd.
Creative Quality Alignment: Expert Tacit Knowledge Transfer via Chain-of-Thought Fine-Tuning
May 25, 2026Bo Zou, Chao XuFine-Tuning LLM Alignment
Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization
May 25, 2026Meshal Alamr, Hassan Alqaeri, Abdullah AldahlawiFine-Tuning Arabic NLP
Thaka, Advanced AI and Information Technology Riyadh, Saudi Arabia
Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa
May 25, 2026Pritesh JhaFine-Tuning PII Detection
Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts
May 24, 2026Xiangtian Ji, Yuxin Chen, Zhengzhou Cai +3Fine-Tuning LLM Interpretability
National University of Singapore · Beijing University of Posts and Telecommunications · University of Science and Technology of China
ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
May 20, 2026Yongkang Liu, Zijing Wang, Mengjie Zhao +7Fine-Tuning Memory-Efficient Fine-Tuning
Northeastern University, China · Shanghai Jiao Tong University, China · CIS, LMU Munich, Germany +2
An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees
May 19, 2026Hoang Tran, Jorge Ramirez, Jiayi Wang +3Fine-Tuning Privacy-Preserving ML
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831, USA · Departamento de Matem´aticas, Universidad Nacional de Colombia, Medell´ın, Colombia. · Computational Science and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831, USA +1
D-CLING: Prior-Preserving Depth-Conditioned Fine-Tuning for Navigation Foundation Models
May 19, 2026Shintaro Nakaoka, Takayuki Kanai, Kazuhito TanakaVisual Navigation Fine-Tuning
The authors are with the Frontier Research Center, Toyota Motor Corporation, Toyota, Aichi, Japan.
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
May 18, 2026Ali Zindari, Rotem Mulayoff, Sebastian U. StichFine-Tuning Low-Rank Adaptation
Universität des Saarlandes · CISPA Helmholtz Center for Information Security
Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic
May 18, 2026Thomas Sommariva, Francesca Morandi, Simone Calderara +1Fine-Tuning Task Arithmetic
ImageLab, Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
DP-SelFT: Differentially Private Selective Fine-Tuning for Large Language Models
May 17, 2026Haichao Sha, Zihao Wang, Yuncheng Wu +2Fine-Tuning Privacy-Preserving Language Models
Renmin University of China · Nanyang Technological University
Closing the Gap at CRAC 2026: Two-Stage Adaptation for LLM-Based Multilingual Coreference Resolution
May 16, 2026Antoine Bourgois, Olga Seminck, Thierry PoibeauMultilingual Language Models Coreference Resolution
Lattice (CNRS UMR 8094 & ENS-PSL & Universit´e Sorbonne Nouvelle), Montrouge, France

Jun 4, 2026, cs.LGJ/K move · Enter open · S save

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

O. Duranthon, F. Boncoraglio, L. Zdeborová

Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne

Fine-Tuning Low-Rank Adaptation Attention Mechanisms

Open paper AskPDF arXiv

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a solvable framework in which a single-head attention layer is first pre-trained on a data-abundant task and subsequently adapted via a rank-one LoRA update on limited data. In the high-dimensional limit, both stages admit a sharp asymptotic characterization in terms of a finite set of order parameters, yielding explicit predictions for test errors and representation alignment. Our analysis shows that the impact of pre-training on LoRA is summarized by an effective noise term, from which we derive prescriptions for the optimal pre-training procedure. We also demonstrate a regime with a mismatch between the value of the test error and representation quality, and propose an application of our theory to active fine-tuning.