LLM Fine-Tuning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 83% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 509

All topics

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
Oct 8, 2026Zhiqiang Pang, Zihong Sun, Qi Xie +3LLM Fine-Tuning Selective Fine-Tuning
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China
SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning
Oct 8, 2026Kenan Tang, Andong Hua, Chengxuan Qian +2LLM Fine-Tuning Catastrophic Forgetting
University of California, Santa Barbara
Large Language Model-Assisted Preparation of Transportation Management Plans: A Case Study with WisDOT WisTMP System
Oct 7, 2026Zihao Sheng, Pei Li, Zilin Huang +6LLM Fine-Tuning
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI 53706, USA · Department of Civil and Architectural Engineering and Construction Management, University of Wyoming, WY 82071, USA · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA
A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
Oct 7, 2026Jonghyun Han, Younghoon Song, Jongyoul ParkCatastrophic Forgetting Continual Learning for LLMs
Seoul National University of Science and Technology · Korea Institute of Land & Infrastructure Safety Technology
The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs
Oct 7, 2026Jiachen Zhao, Zhengxuan Wu, David Bau +1LLM Fine-Tuning LLM Alignment
Northeastern University · Stanford University
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Oct 6, 2026Vedant Palit, Florent Draye, Nicolas Zucchet +2Memorization in Language Models LLM Fine-Tuning
MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3
FinVector-Market-4B: A Controlled Study of LoRA Adaptation for Structured Financial Tasks
Oct 6, 2026Alina KhaybullinaDomain Adaptation for LLMs LLM Fine-Tuning
Independent Researcher
Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes
Oct 6, 2026Avichal Sahai, Nishant Raj, Animesh SrivastavaLLM Quantization Ternary Quantization
Ofbusiness
HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior
Oct 6, 2026Jin Huang, Diego Ferreras Garrucho, Yutong Xie +4LLM Evaluation LLM Fine-Tuning
University of Michigan, Ann Arbor · London School of Economics · MobLab Inc +1
HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning
Oct 5, 2026Ruiheng Wang, Yubo Hou, Yakun Zhu +3Fine-Tuning LLM Fine-Tuning
Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · VinUniversity, Hanoi, Vietnam
ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement
Oct 5, 2026Xingbo Yao, Xiaoman Wang, Zhengwu Lei +11LLM Fine-Tuning LLM Post-Training
HKUST(GZ) · Xiaohongshu Inc. · ECNU +3
Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training
Oct 5, 2026Andrew Geyko, Marius Mosbach, André BrinkmannOn-Device Training LLM Fine-Tuning
Saarland University Saarbrücken, Germany
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-Tuning LLM Fine-Tuning
Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Turnslide: Scalable Multi-Turn Data Synthesis by Walking a Finite-State Machine
Oct 5, 2026Aaron Fainman, Gabriela Kadlecová, Maciej Gryka +7LLM Fine-Tuning Synthetic Data Generation
Imperial College London · distil labs · Agon +1
VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges
Oct 4, 2026Qiushui Xu, Syamil Mohd Razak, Tao Yuan +1LLM-as-a-Judge LLM Fine-Tuning
Penn State University · Amazon
Robust Parameter-Efficient LLM Adaptation on Analog Hardware
Oct 4, 2026Jindan Li, Zhaoxian Wu, Tianyi ChenCompute-in-Memory LLM Fine-Tuning
Cornell University
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Oct 1, 2026Jichao Jiang, Cristian McGee, El Houcine Bergou +2Memory-Efficient Fine-Tuning LLM Fine-Tuning
University of Central Florida · Mohammed VI Polytechnic University
Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
Oct 1, 2026Cristian McGee, El Houcine Bergou, Aritra DuttaZeroth-Order Optimization LLM Fine-Tuning
University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Oct 1, 2026Yinheng Li, Justin WagleLLM Inference LLM Fine-Tuning
Microsoft
No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
Oct 1, 2026Lewis MitchellLLM Fine-Tuning Model Collapse
Adelaide Data Science Centre School of Mathematical Sciences Adelaide University Adelaide SA 5005, Australia
TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
Oct 1, 2026Fengpeng Li, Kemou Li, Qizhou Wang +3LLM Fine-Tuning Adversarial Attacks on LLMs
PRADA Lab, King Abdullah University of Science and Technology · State Key Laboratory of Internet of Things for Smart City, University of Macau · Imperfect Information Learning Team, RIKEN Center for Advanced Intelligence Project +1
RPTune: Learned Context Curation for LLM Catalog Search
Oct 1, 2026Chuxuan Hu, Hejie Cui, Norman Huang +3In-Context Retrieval LLM Fine-Tuning
Google · University of Illinois Urbana-Champaign
Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations
Oct 1, 2026Jungwon Park, Changin Choi, Jimyeong Kim +2LLM Fine-Tuning In-Context Learning
RICS · Daegu Gyeongbuk Institute of Science and Technology · IPAI +3
Training-Aware Target Coverage for Synthetic Data Selection
Sep 30, 2026Yang Ba, Michelle V. Mancenido, Rong PanLLM Fine-Tuning Data Selection
School of Computing and Augmented Intelligence, Arizona State University · School of Mathematical and Natural Sciences, Arizona State University
Towards Robust Numerical Claim Verification
Sep 30, 2026Peter Røysland Aarnes, Vinay SettyNumerical Reasoning in Language Models Claim Verification
University of Stavanger · Factiverse AI and University of Stavanger
Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow
Sep 30, 2026Polina Tsvilodub, Andreas Waldis, Linlu Qiu +2Fine-Tuning LLM Fine-Tuning
University of Tübingen · Massachusetts Institute of Technology · New York University
Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning
Sep 30, 2026Hongyu Chen, Xinyi Luo, Ming Zhao +6LLM Fine-Tuning
Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China
A helps B while B hurts A: directed transfer in instruction-tuning mixture
Sep 30, 2026Nima H. Siboni, Vahid RostamiLLM Fine-Tuning Instruction-Tuning Data Selection
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany · Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Germany
QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
Sep 30, 2026Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichLLM Fine-Tuning Domain-Adaptive Pretraining
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2LLM Alignment LLM Fine-Tuning
EleutherAI

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Zhiqiang Pang, Zihong Sun, Qi Xie, Jun Shu+2

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China

LLM Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting Low-Rank Adaptation

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.

Momentum

Related topics

Latest papers 509