LLM Fine-Tuning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 83% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 509

All topics

Omega-S: A Functional Resilience Index for LLM Fine-Tuning
Aug 4, 2026Alberto AcedoContinual Learning for LLMs LLM Fine-Tuning
Biome Makers Inc.
Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Study
Aug 4, 2026Ahmed Amine Aliane, Nasredine Semmar, Hassina AlianeMultilingual Language Models LLM Fine-Tuning
Arabic Institute for Translation, Algiers, Algeria · CEA LIST, Paris-Saclay, France · CERIST, Algeria
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models
Aug 4, 2026Lele Zheng, Weifeng Kong, Xinyi Zhang +3Memory-Efficient Fine-Tuning Privacy-Preserving Language Models
School of Computer Science and Technology, Xidian University
MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
Aug 4, 2026Yiming Zeng, Lei Lu, Zexin Li +10LLM Fine-Tuning Low-Rank Adaptation
University of Connecticut · Northeastern University · Nanyang Technological University +5
LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
Aug 4, 2026Linhan Xia, Rui Liu, Zhaofeng Zhang +3Fine-Tuning LLM Fine-Tuning
University of Oklahoma · 4Tencent · 2Imperial College London +5
Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
Aug 3, 2026Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim +1Zeroth-Order Optimization LLM Fine-Tuning
Chosun University · Korea University Sejong Campus
Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection
Jul 31, 2026Chaimae Abouzahir, Musa Khan, Hala Ali-Hassan +7LLM Fine-Tuning Low-Rank Adaptation
New York University Abu Dhabi · Cleveland Clinic Abu Dhabi
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
Jul 31, 2026Guanzhi Deng, Haibo Wang, Kuan Wu +5Mixture-of-Experts Language Models Mixture of Experts
City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · University of Waterloo, Ontario, Canada +1
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits
Jul 31, 2026Namkyung Yoon, Sanghong Kim, Hwangnam KimContinual Learning for LLMs LLM Fine-Tuning
School of Electrical Engineering, Korea University Seoul 02841, Republic of Korea
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
Jul 31, 2026Mark ShapiroLLM Fine-Tuning Recurrent Transformers
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5LLM Serving LLM Fine-Tuning
McGill University · Canada · MBZUAI +3
Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Games
Jul 30, 2026Joshua Caiata, Sreepriya Pulyassary, Xiang Li +1LLM Fine-Tuning Strategic Reasoning Risks in Language Models
University of Waterloo
MedLLM: An Open Medical Language Model at the Sub-Billion Scale
Jul 29, 2026Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh +1LLM Fine-Tuning Small Language Models
German Research Center for Artificial Intelligence (DFKI), Germany · Saarland University, Germany
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
Jul 29, 2026Ru Peng, Tianyu Zhao, Xijun Gu +9LLM Fine-Tuning Instruction Tuning
Zhejiang University · Inclusion AI, Ant Group · Ant Group +2
MemSFT: Mitigating Alignment Tax with an External Parametric Memory
Jul 28, 2026Jiarui Wang, Xiang Shi, Jiaqi Cao +8LLM Alignment Memory-Augmented Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · School of Computer Science, Shanghai Jiao Tong University +1
From Semantics to Readout: Mechanistic Understanding of Audio Tokens after Fine-Tuning for Temporal Audio Grounding
Jul 28, 2026Yujian Ma, Jinqiu Sang, Ruizhe Li +2LLM Grounding LLM Fine-Tuning
1Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China · School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK +1
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Jul 28, 2026Yuan Zhang, Jiang Hu, Zhijian Lai +2Stiefel Manifold Optimization Riemannian Optimization
Center for Data Science, Peking University, Beijing 100871, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China +2
Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe
Jul 28, 2026Chaemin Jang, Dongman Lee, Jihee KimLanguage Model Decoding LLM Prompting
School of Computing, KAIST · School of Business and Technology Management, KAIST
The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
Jul 26, 2026Peng XieLLM Fine-Tuning Low-Rank Adaptation
Technical University of Munich, Campus Heilbronn
Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
Jul 26, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Grounding
Department of Electrical and Computer Engineering, North South University Dhaka, Bangladesh · Department of Computer Science and Engineering, BRAC University Dhaka, Bangladesh
BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi
Jul 25, 2026Hariom Ingle, Ronit Ghode, Ishwari Gondkar +2Named Entity Recognition LLM Evaluation
Department of Information Technology, PICT, Pune, India · L3Cube Labs, Pune, India · Indian Institute of Technology Madras, Chennai, India
IKS-Instruct: A 24,000-Example Multilingual Dataset for Teaching Language Models Indian Knowledge Systems
Jul 25, 2026Shwetha Singaravelu, Gayathri Muruganantham, Lakshmi Rajendran +1LLM Fine-Tuning Cultural Knowledge in Language Models
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi, New Delhi 110016, India · RSL Quantum, FITT, IIT Delhi, New Delhi 110016, India
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China
Benchmarking Fine-tuning and Retrieval Strategies for a Multimodal Language Model on the NRC Reactor Operator Licensing Examination
Jul 24, 2026Isak Hwang, Yoon Pyo LeeRetrieval-Augmented Generation LLM Fine-Tuning
aDepartment of Nuclear Engineering, Hanyang University, 222 Wangsimni-ro, 04763, Seongdong-gu, Seoul, South Korea · bThe Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA
Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
Jul 23, 2026Joan Figuerola HurtadoKnowledge Augmentation for Language Models LLM Fine-Tuning
Independent Researcher
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Jul 23, 2026Mohammed Suhail B NadafLLM Fine-Tuning Emergent Misalignment in Language Models
Independent
news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling
Jul 23, 2026Pascal Stolzenburg, Jonas Golde, Max Dallabetta +1LLM Fine-Tuning Document Information Extraction
Humboldt-Universität zu Berlin
LKValues: Aligning Large Language Models with Sri Lankan Societal Values
Jul 22, 2026Nethmi Muthugala, Supryadi, Surangika Ranathunga +7Multilingual Language Model Evaluation LLM Alignment
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka +2
Statistical Inference for Rank Allocation in Low-Rank Adaptation
Jul 22, 2026Yihang Gao, Vincent Y. F. TanLLM Fine-Tuning Low-Rank Adaptation
Department of Mathematics, National University of Singapore · Department of Mathematics and Department of Electrical and Computer Engineering, National University of Singapore
When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
Jul 21, 2026Junyi Sha, Renfei Tan, David Simchi-LeviLLM Fine-Tuning Imitation Learning
Institute for Data, Systems, and Society MIT Cambridge, MA 02139 · Department of Civil and Environmental Engineering MIT Cambridge, MA 02139

Aug 4, 2026, cs.LGJ/K move · Enter open · S save

Omega-S: A Functional Resilience Index for LLM Fine-Tuning

Alberto Acedo

Biome Makers Inc.

Continual Learning for LLMs LLM Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

Momentum

Related topics

Latest papers 509