LLM Fine-Tuning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

76 papers in the last four weeks, up 29% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 509

All topics

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
May 6, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia
Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs
May 6, 2026Santosh Premi Adhikari, Radu Timofte, Dmitry IgnatovLarge Language Model-Guided NAS LLM Fine-Tuning
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning
May 6, 2026Xiao Wang, Yifei Zhang, YongKang Liu +4LLM Fine-Tuning LLM Safety
School of Computer Science and Engineering, Northeastern University, Shenyang, China · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao, China
On Adaptivity in Zeroth-Order Optimization
May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order Optimization LLM Fine-Tuning
Qualcomm AI Research
Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF
May 5, 2026Mullosharaf K. ArabovLLM Fine-Tuning Low-Resource Language Processing
KAZAN (VOLGA REGION) FEDERAL UNIVERSITY · INSTITUTE OF COMPUTATIONAL MATHEMATICS AND INFORMATION TECHNOLOGIES
Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
May 5, 2026Mullosharaf K. ArabovLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT
May 5, 2026Mahmoud Hanouneh, Radu Timofte, Dmitry IgnatovLLM Fine-Tuning Automated Machine Learning
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
May 5, 2026Yihang Peng, Peng Jin, Jie Gong +4LLM Fine-Tuning Low-Rank Adaptation
School of Computer Science and Software Engineering, Southwest Petroleum University · School of Electronic Information and Artificial Intelligence, Leshan Normal University
Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
May 5, 2026Zhe Li, Bicheng Ying, Zidong Liu +1Zeroth-Order Optimization LLM Fine-Tuning
Department of Computing and Information Sciences Ph.D., Rochester Institute of Technology, Rochester, NY, USA · Google Inc., Los Angeles, CA, USA · ComboCurve Inc., Houston, TX, USA
The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It
May 5, 2026Gabriel GarciaNumerical Reasoning in Language Models Transformer Interpretability
Independent Researcher
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language Models LLM Fine-Tuning
University of Michigan
Self-Mined Hardness for Safety Fine-Tuning
May 4, 2026Prakhar Gupta, Garv Shah, Donghua ZhangHard Negative Mining LLM Fine-Tuning
University of Michigan
Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM
May 3, 2026Luo Ji, Qi Qin, Ningyuan Xi +3Meta-Learning LLM Fine-Tuning
Geely AI Lab, Geely Auto Group, Zhejiang, China · Peking University, Beijing, China
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
May 3, 2026Sadia Asif, Mohammad Mohammadi AmiriFine-Tuning LLM Fine-Tuning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA
Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
May 3, 2026Nikolaos Giarelis, Charalampos Mastrokostas, Nikos KaracapilidisLLM Fine-Tuning Question Answering
Industrial Management and Information Systems Lab, MEAD University of Patras, Rio Patras, Greece
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
May 2, 2026Zhiwen Ruan, Yichao Du, Jianjie Zheng +6LLM Fine-Tuning Large Language Model-Guided Optimization
Southern University of Science and Technology · Alibaba Group · Shanghai University of Finance and Economics +2
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
May 1, 2026Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu KaoLLM Fine-Tuning Low-Rank Adaptation
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Department of Computer Science, National Tsing Hua University
LLM-assisted gNB Parameter Configuration for Radio Access Network
May 1, 2026Yao-Cong Dong, Maria Amparo Canaveras Galdon, Ari Uskudar +3LLM Fine-Tuning Wireless Communications
Dept. of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taiwan · NVIDIA, USA · Colorado State University, USA
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
May 1, 2026Zhijie Cai, Haolong Chen, Guangxu ZhuMemory-Efficient Fine-Tuning Zeroth-Order Optimization
Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong-Shenzhen · Shenzhen Loop Area Institute
RadLite: Multi-Task LoRA Fine-Tuning of Small Language Models for CPU-Deployable Radiology AI
May 1, 2026Pankaj Gupta, Kartik BoseLLM Fine-Tuning Small Language Models
Postgraduate Institute of Medical Education and Research, Chandigarh, India ∗Corresponding author: Additional Professor, Department of Radiodiagnosis and Imaging, Postgraduate Institute of Medical Education and Research, Chandigarh, India · Postgraduate Institute of Medical Education and Research, Chandigarh, India
XekRung Technical Report
Apr 30, 2026Jiutian Zeng, Junjie Li, Chengwei Dai +13LLM Fine-Tuning Cybersecurity
Alibaba Security AGI Lab
Diagnosing Capability Gaps in Fine-Tuning Data
Apr 30, 2026Saeid Asgari Taghanaki, Rakshanda Agarwal, Bruce Sun +10Fine-Tuning LLM Fine-Tuning
Microsoft
Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoors
Apr 30, 2026Zi Li, Tian Zhou, Wenze Li +3Data Leakage LLM Backdoor Attacks
Nanjing University
ChipLingo: A Systematic Training Framework for Large Language Models in EDA
Apr 30, 2026Lei Li, Xingwen Yu, Jianguo Ni +4Knowledge Augmentation for Language Models LLM Fine-Tuning
Ickylin AI Team
BoostLoRA: Growing Effective Rank by Boosting Adapters
Apr 30, 2026Raviteja Anantha, Nick Levato, Layne C. PriceLLM Fine-Tuning Adapter Tuning
Amazon
Efficient Training on Multiple Consumer GPUs with RoundPipe
Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2GPU Acceleration LLM Fine-Tuning
Tsinghua University Beijing, China
MoRFI: Monotonic Sparse Autoencoder Feature Identification
Apr 29, 2026Dimitris Dimakopoulos, Shay B. Cohen, Ioannis KonstasLLM Hallucination Mitigation LLM Interpretability
University of Edinburgh, UK · Heriot-Watt University, UK
Domain-Adapted Small Language Models for Reliable Clinical Triage
Apr 29, 2026Manar Aljohani, Brandon Ho, Kenneth McKinley +2Clinical Triage LLM Fine-Tuning
Department of Computer Science, Virginia Tech, Blacksburg, VA, US · Children’s National Hospital, Washington DC, USA
TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
Apr 29, 2026Jinho Choo, JunSeung Lee, Jimyeong Kim +3Multilingual Language Models RL for Language Models
Samsung SDS
SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
Apr 29, 2026Yimeng Shan, Zhaorui Zhang, Sheng Di +3Communication-Efficient Distributed Training LLM Fine-Tuning
Hong Kong Polytechnic University Hong Kong · Argonne National Labratory USA · University of California, Merced USA +1

May 6, 2026, cs.CLJ/K move · Enter open · S save

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

Mullosharaf K. Arabov, Svetlana S. Khaybullina

Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia

LLM Fine-Tuning Low-Resource Language Processing Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language, a low-resource agglutinative language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.

Momentum

Related topics

Latest papers 509