LLM Fine-Tuning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 83% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 509

All topics

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language Models LLM Fine-Tuning
University of Michigan
Self-Mined Hardness for Safety Fine-Tuning
May 4, 2026Prakhar Gupta, Garv Shah, Donghua ZhangHard Negative Mining LLM Fine-Tuning
University of Michigan
Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM
May 3, 2026Luo Ji, Qi Qin, Ningyuan Xi +3Meta-Learning LLM Fine-Tuning
Geely AI Lab, Geely Auto Group, Zhejiang, China · Peking University, Beijing, China
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
May 3, 2026Sadia Asif, Mohammad Mohammadi AmiriFine-Tuning LLM Fine-Tuning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA
Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
May 3, 2026Nikolaos Giarelis, Charalampos Mastrokostas, Nikos KaracapilidisLLM Fine-Tuning Question Answering
Industrial Management and Information Systems Lab, MEAD University of Patras, Rio Patras, Greece
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
May 2, 2026Zhiwen Ruan, Yichao Du, Jianjie Zheng +6LLM Fine-Tuning Large Language Model-Guided Optimization
Southern University of Science and Technology · Alibaba Group · Shanghai University of Finance and Economics +2
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
May 1, 2026Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu KaoLLM Fine-Tuning Low-Rank Adaptation
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Department of Computer Science, National Tsing Hua University
LLM-assisted gNB Parameter Configuration for Radio Access Network
May 1, 2026Yao-Cong Dong, Maria Amparo Canaveras Galdon, Ari Uskudar +3LLM Fine-Tuning Wireless Communications
Dept. of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taiwan · NVIDIA, USA · Colorado State University, USA
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
May 1, 2026Zhijie Cai, Haolong Chen, Guangxu ZhuMemory-Efficient Fine-Tuning Zeroth-Order Optimization
Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong-Shenzhen · Shenzhen Loop Area Institute
RadLite: Multi-Task LoRA Fine-Tuning of Small Language Models for CPU-Deployable Radiology AI
May 1, 2026Pankaj Gupta, Kartik BoseLLM Fine-Tuning Small Language Models
Postgraduate Institute of Medical Education and Research, Chandigarh, India ∗Corresponding author: Additional Professor, Department of Radiodiagnosis and Imaging, Postgraduate Institute of Medical Education and Research, Chandigarh, India · Postgraduate Institute of Medical Education and Research, Chandigarh, India
XekRung Technical Report
Apr 30, 2026Jiutian Zeng, Junjie Li, Chengwei Dai +13LLM Fine-Tuning Cybersecurity
Alibaba Security AGI Lab
Diagnosing Capability Gaps in Fine-Tuning Data
Apr 30, 2026Saeid Asgari Taghanaki, Rakshanda Agarwal, Bruce Sun +10Fine-Tuning LLM Fine-Tuning
Microsoft
Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoors
Apr 30, 2026Zi Li, Tian Zhou, Wenze Li +3Data Leakage LLM Backdoor Attacks
Nanjing University
ChipLingo: A Systematic Training Framework for Large Language Models in EDA
Apr 30, 2026Lei Li, Xingwen Yu, Jianguo Ni +4Knowledge Augmentation for Language Models LLM Fine-Tuning
Ickylin AI Team
BoostLoRA: Growing Effective Rank by Boosting Adapters
Apr 30, 2026Raviteja Anantha, Nick Levato, Layne C. PriceLLM Fine-Tuning Adapter Tuning
Amazon
Efficient Training on Multiple Consumer GPUs with RoundPipe
Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2GPU Acceleration LLM Fine-Tuning
Tsinghua University Beijing, China
MoRFI: Monotonic Sparse Autoencoder Feature Identification
Apr 29, 2026Dimitris Dimakopoulos, Shay B. Cohen, Ioannis KonstasLLM Hallucination Mitigation LLM Interpretability
University of Edinburgh, UK · Heriot-Watt University, UK
Domain-Adapted Small Language Models for Reliable Clinical Triage
Apr 29, 2026Manar Aljohani, Brandon Ho, Kenneth McKinley +2Clinical Triage LLM Fine-Tuning
Department of Computer Science, Virginia Tech, Blacksburg, VA, US · Children’s National Hospital, Washington DC, USA
TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
Apr 29, 2026Jinho Choo, JunSeung Lee, Jimyeong Kim +3Multilingual Language Models RL for Language Models
Samsung SDS
SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
Apr 29, 2026Yimeng Shan, Zhaorui Zhang, Sheng Di +3Communication-Efficient Distributed Training LLM Fine-Tuning
Hong Kong Polytechnic University Hong Kong · Argonne National Labratory USA · University of California, Merced USA +1
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China
EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation
Apr 28, 2026Ting-Wei Li, Sirui Chen, Jiaru Zou +4LLM Fine-Tuning Synthetic Data Generation
University of Illinois Urbana-Champaign
G-Loss: Graph-Guided Fine-Tuning of Language Models
Apr 28, 2026Aditya Sharma, Vinti Agarwal, Rajesh KumarLLM Fine-Tuning Text Classification
BITS Pilani, India · Bucknell University, USA
Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data
Apr 28, 2026Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani +1Social Media Analysis LLM Fine-Tuning
Institut Teknologi Sumatera Lampung Selatan, Indonesia
LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model
Apr 28, 2026Youngjoon Jang, Chanhee Park, Hyeonseok Moon +5Legal NLP LLM Fine-Tuning
Department of Computer Science and Engineering, Korea University · KT (Korea Telecom)
BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate
Apr 28, 2026Arnon Mazza, Elad LeviLLM Guardrails LLM Fine-Tuning
Plurai Inc.
From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models
Apr 28, 2026Ling Shi, Xinwei Wu, Xiaohu Zhao +7LLM Fine-Tuning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Alibaba Group, China
Contextual Linear Activation Steering of Language Models
Apr 27, 2026Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Language Model Steering LLM Fine-Tuning
UC San Diego · MIT · Broad Institute of MIT and Harvard
Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study
Apr 27, 2026Sivajeet Chand, Kevin Nguyen, Peter Kuntz +1LLM Fine-Tuning Code Generation
Technical University of Munich · Munich, Germany · BMW Group
A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations
Apr 27, 2026Zihan Liu, Yizhen Wang, Rui Wang +2Privacy-Preserving Language Models LLM Fine-Tuning
Zhejiang University, Ningbo, China

May 4, 2026, cs.CLJ/K move · Enter open · S save

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

Prakhar Gupta, Garv Shah, Satyam Goyal, Anirudh Kanchi

University of Michigan

Memory-Augmented Language Models LLM Fine-Tuning Sparse Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.

Momentum

Related topics

Latest papers 509