Fine-Tuning

Follow this topic Ask about this topic

Latest papers 205

31 papers in the last four weeks, up 72% on the four weeks before. 0.2% of all new papers.

All topics

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Apr 24, 2026Haoze He, Xingyuan Ding, Xuan Jiang +5Supervised Fine-Tuning Fine-Tuning
Carnegie Mellon University · MIT · UCSD +1
Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learning
Apr 23, 2026Paul-Tiberiu Iordache, Elena Burceanu, Mihai DascaluFine-Tuning Continual Learning
Bitdefender, Romania · Politehnica University of Bucharest, Romania
Compatibility-Aware Dynamic Fine-Tuning for Large Language Models
Apr 22, 2026Yucheng Zhou, Junwei Sheng, Qianning Wang +1Supervised Fine-Tuning Fine-Tuning
SKL-IOTSC, CIS, University of Macau · Auckland University of Technology
IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning
Apr 22, 2026Wenjie Liao, Like Wu, Liangjie Zhao +2Fine-Tuning LLM Fine-Tuning
Graduate School of Information, Production and Systems, Waseda University, Fukuoka, Japan · Institute of Computing Technology, University of the Chinese Academy of Sciences
Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
Apr 21, 2026Xin Ning, Qiankun Li, Xiaolong Huang +5Fine-Tuning Neural Network Optimization
Institute of Semiconductors, Chinese Academy of Sciences, Beijing, 10083, China · College of Computing and Data Science (CCDS), Nanyang Technological University, 639798, Singapore · Mila-Quebec AI Institute, Montreal, QC H2S 3H1, Canada +3
Hierarchically Robust Zero-shot Vision-language Models
Apr 20, 2026Junhao Dong, Yifei Zhang, Hao Zhu +2Hierarchical Classification VLM Robustness
1Nanyang Technological University · 2CFAR, IHPC, A*STAR · 3Northwest Polytechnical University +2
Evolutionary fine tuning of quantized convolution-based deep learning models
Apr 19, 2026Marcin PietrońFine-Tuning Convolutional Neural Networks
AGH University Kraków, Poland
Decomposing the Depth Profile of Fine-Tuning
Apr 19, 2026Jayadev BillaFine-Tuning Representation Drift
MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation
Apr 18, 2026Bo Li, Ningyuan Deng, Tianyu Dong +3Fine-Tuning VLM Adaptation
School of Software, Tsinghua University, Beijing, China · Baidu Inc., Beijing, China · School of Information Resource Management, Renmin University of China,Beijing, China +2
Not All Forgetting Is Equal: Retention Dynamics in Fine-Tuned Image Classifiers
Apr 13, 2026Miit Daga, Swarna Priya RamuFine-Tuning Data Selection
School of Computer Science Engineering and Information Systems, Vellore Institute of Technology, Vellore, 632014, Tamil Nadu, India
Where Does Robustness Live? Neuron-Guided Adaptation for Retrieval-Augmented Language Models
Apr 2, 2026Jae O Lee, Jaemin Kim, Sumyeong Ahn +1Retrieval-Augmented Generation Fine-Tuning
1Hanyang University · 2KENTECH
Retrieval-Augmented LLM Agents: Learning to Learn from Experience
Mar 18, 2026Thomas Palmeira Ferraz, Romain Deffayet, Vassilina Nikoulina +2Fine-Tuning LLM Agent Training
NAVER LABS Europe, Meylan, France
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
Mar 17, 2026Ruijia Yang, Zeyi WenFine-Tuning Memory-Efficient Fine-Tuning
HKUST(GZ) Guangzhou, China · HKUST Guangzhou, China
Not All Invariants Are Equal: Curating Training Data to Accelerate Program Verification with SLMs
Mar 16, 2026Ido Pinto, Yizhak Yisrael Elboher, Haoze Wu +2Fine-Tuning Training Data Curation
Hebrew University of Jerusalem, Israel · Amherst Col · VMware Research by Broadcom, USA.
A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
Feb 23, 2026Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4Neural Network Generalization Representation Learning
Department of Engineering, University of Cambridge · Institute of Science and Technology, Austria (ISTA) · Gatsby Computational Neuroscience Unit and Sainsbury Wellcome Centre, UCL +1
Selective Fine-Tuning for Targeted and Robust Concept Unlearning
Feb 8, 2026Mansi, Avinash Kori, Francesca Toni +1Fine-Tuning Adversarial Robustness
Department of Computing Imperial College London, United Kingdom
Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions
Jan 20, 2026Fan Huang, Haewoon Kwak, Jisun AnLLM Evaluation Fine-Tuning
Indiana University Bloomington, United States
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2
Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
Dec 28, 2025Mingyuan Zhang, Yue Bai, Yifan Wang +2Fine-Tuning VLM Adaptation
College of Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University
FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning
Nov 22, 2025Bo Yin, Xiaobin Hu, Xingyu Zhou +7Fine-Tuning Adaptive Model Routing
National University of Singapore · University of Electronic Science and Technology of China · Nanyang Technological University +4
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Sep 29, 2025Xin Qiu, Yulu Gan, Conor F. Hayes +6Fine-Tuning Evolutionary Optimization
Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1
When few labeled target data suffice: a theory of semi-supervised domain adaptation via fine-tuning from multiple adaptive starts
Jul 19, 2025Wooseok Ha, Yuansi ChenFine-Tuning Domain Adaptation
Department of Mathematical Sciences, KAIST · Department of Mathematics, ETH Zürich
GenTL: A General Transfer Learning Model for Building Thermal Dynamics
Jan 23, 2025Fabian Raisch, Thomas Krug, Christoph Goebel +1Fine-Tuning Transfer Learning
Technical University of Applied Sciences Rosenheim Rosenheim, Germany · Technical University of Munich Munich, Germany
Optimizing Three Critical Factors for Practical and Effective OOD Detection Fine-Tuning
Aug 2, 2023Hyunjun Choi, JaeHo Chung, Hawook JeongFine-Tuning OOD Detection
ASRI, ECE., Seoul National University, South Korea · RideFlux Inc., South Korea
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Date pendingXianming Li, Zongxi Li, Tsz-fung Andrew Lee +3Fine-Tuning Adapter Tuning
PolyU · School of Data Science, Lingnan University, Hong Kong SAR

Apr 24, 2026, cs.LGJ/K move · Enter open · S save

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou+4

Carnegie Mellon University · MIT · UCSD+1

Supervised Fine-Tuning Fine-Tuning Long-Tail Learning Sparse Mixture-of-Experts Mixture-of-Experts Models

Open paper AskPDF arXiv

Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMixer and ESFT mitigate router collapse with dense mixing or auxiliary load-balancing losses, but these introduce noisy gradients that often degrade performance. In preliminary experiments, we systematically pruned experts and observed that while certain super experts are activated far more frequently, discarding less used experts still leads to notable performance degradation. This suggests that even rarely activated experts encode non-trivial knowledge useful for downstream tasks. Motivated by this, we propose an auxiliary-loss-free MoE SFT framework that combines bias-driven sparsification with always-active gated condenser experts. Rather than enforcing balanced activation across all experts, our method encourages task-relevant experts to remain active while pushing long-tailed experts toward inactivity. The condenser experts provide a persistent, learnable pathway that alleviates gradient starvation and facilitates consolidation of information that would otherwise remain fragmented across sparsely activated experts. Analysis further suggest that this design better preserves long-tailed expert information under sparse routing. Experiments on large-scale MoE models demonstrate that our approach outperforms state-of-the-art SFT baselines such as DenseMixer and ESFT, achieving average gain of 2.5%+ on both mathematical reasoning and commonsenseQA benchmarks.