LLM Fine-Tuning

LLM: Large Language Model

Ask about this topic

Overview

LLM fine-tuning studies how additional training can adapt a pretrained model to new tasks, specialized knowledge, languages, or desired behaviors. Common approaches include supervised fine-tuning on curated or synthetic examples, preference optimization and reinforcement learning, and parameter-efficient methods such as LoRA and QLoRA that update only a small part of the model. Researchers also develop data-selection methods and selective layer or neuron updates to reduce training cost, and evaluate results on settings such as reasoning, code generation, multilingual tasks, and domain-specific question answering. A central challenge is gaining reliable task performance without losing general abilities or safety; results depend on the data, model, and evaluation, and retrieval can sometimes be a better alternative for adding knowledge.

Latest papers

CardsList

758 papers76 papers in the last four weeks, up 15% on the four weeks before. 0.4% of all new papers.

Apr 21, 2026

Fine-Tuning Small Reasoning Models for Quantum Field Theory

Despite the growing application of large language models (LLMs) to theoretical physics, there has been little academic exploration of how domain-specific physics reasoning ability develops during training. To investigate this, we perform the first academic comparison of reinforcement learning and supervised fine-tuning for LLMs on verifiable problem-solving tasks in theoretical physics. Because the open-source verifiable training data required to develop such capabilities is scarce, we built a robust data generation pipeline that can both create synthetic problems and make existing human-authored problems suitable for model training. Selecting quantum field theory (QFT) as our primary domain, we generated over 2,500 synthetic problems alongside a curated collection of human-adapted problems sourced from arXiv and standard pedagogical resources. We conduct both reinforcement learning (RL) and supervised fine-tuning (SFT) experiments, benchmarking performance gains as well as generalization to other physics domains. Although both methods improve over the base model on every dataset, SFT gains are concentrated on in-distribution synthetic tasks, while RL generalizes better, outperforming SFT on the human-adapted datasets and TPBench. We perform an extensive analysis of model chain-of-thought (CoT) reasoning before and after fine-tuning to understand how reasoning errors evolve during RL and SFT. Finally, we publicly release our data pipeline, verifiable QFT training data, and ~200M tokens of QFT reasoning traces.

Nathaniel S. Woodward, Zhiqi Gao, Yurii Kvasiuk +3

Department of Physics, University of Wisconsin-Madison · Department of Computer Science, University of Wisconsin-Madison · Perimeter Institute for Theoretical Physics

Physical ReasoningLLM Fine-TuningRL for Language Model ReasoningLarge Reasoning Models
AskPDF
Apr 20, 2026

Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models by relying solely on forward passes. However, its practical adoption is severely constrained by slow wall-clock convergence and high estimation variance. In this work, we dissect the runtime characteristics of ZO algorithms and identify a critical system bottleneck where the generation of perturbations and parameter updates accounts for over 40% of the training latency. We argue that the standard uniform exploration strategy is fundamentally flawed as it fails to account for the heterogeneous sensitivity of layers in deep networks, resulting in computationally wasteful blind searches. To address this structural mismatch, we propose AdaLeZO, an Adaptive Layer-wise ZO optimization framework. By formulating the layer selection process as a non-stationary Multi-Armed Bandit problem, AdaLeZO dynamically allocates the limited perturbation budget to the most sensitive parameters. We further introduce an Inverse Probability Weighting mechanism based on sampling with replacement, which guarantees unbiased gradient estimation while effectively acting as a temporal denoiser to reduce variance. Extensive experiments on LLaMA and OPT models ranging from 6.7B to 30B parameters demonstrate that AdaLeZO achieves 1.7x to 3.0x wall-clock acceleration compared to state-of-the-art methods. Crucially, AdaLeZO functions as a universal plug-and-play module that seamlessly enhances the efficiency of existing ZO optimizers without incurring additional memory overhead.

Fei Wang, Li Shen, Liang Ding +3

South China University of Technology · Shenzhen Campus of Sun Yat-sen University · University of Sydney +2

Zeroth-Order OptimizationLLM Fine-Tuning
AskPDF
Apr 20, 2026

LeGo-Code: Can Modular Curriculum Learning Advance Complex Code Generation? Insights from Text-to-SQL

Recently, code-oriented large language models (LLMs) have demonstrated strong capabilities in translating natural language into executable code. Text-to-SQL is a significant application of this ability, enabling non-technical users to interact with relational databases using natural language. However, state-of-the-art models continue to struggle with highly complex logic, particularly deeply nested statements involving multiple joins and conditions, as well as with real-world database schemas that are noisy or poorly structured. In this paper, we investigate whether curriculum learning can improve the performance of code-based LLMs on Text-to-SQL tasks. Employing benchmarks including Spider and BIRD, we fine-tune models under different curriculum strategies. Our experiments show that naive curriculum, simply ordering training samples by complexity in a single epoch, fails to surpass standard fine-tuning due to catastrophic forgetting. To overcome this, we propose a Modular Adapter Composition (MAC) strategy. By sequentially training tier-specific adapters on incremental complexity levels (Easy to Extra-Hard), we create a scaffolded learning environment that improves performance on complex queries. Our approach not only produces measurable performance gains on the Spider and BIRD benchmarks but also provides a flexible, "Lego-like" architecture, allowing models to be composed and deployed based on specific schema difficulty requirements. These findings demonstrate that structured, modular learning is a superior alternative to monolithic fine-tuning for mastering the syntax and logic of complex code generation.

Salmane Chafik, Saad Ezzini, Ismail Berrada

Mohammed VI Polytechnic University Morocco · King Fahd University of Petroleum and Minerals Saudi Arabia

LLM Fine-TuningCode GenerationText-to-SQL
AskPDF
Apr 20, 2026

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA AA matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the AA matrix is frozen, and only the BB matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Weicheng Lin, Yi Zhang, Jiawei Dang +1

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-TuningLow-Rank AdaptationAdaptive Rank AllocationParameter-Efficient Fine-Tuning
AskPDF
Apr 20, 2026

JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew

Despite significant advances in large language models, personalizing them for individual decision-makers remains an open problem. Here, we introduce a synthetic-organic supervision pipeline that transforms raw judicial decisions into instruction-tuning data, enabling parameter-efficient fine-tuning of personalized models for individual judges in low-resource settings. We compare our approach to state-of-the-art personalization techniques across three different tasks and settings. The results show that Causal Language Modeling followed by synthetically generated instruction-tuning significantly outperforms all other baselines, providing significant improvements across lexical, stylistic, and semantic similarity. Notably, our model-generated outputs are indistinguishable from the reasoning of human judges, highlighting the viability of efficient personalization, even in low-resource settings.

Itay Razumenko, Arnon Sturm, Nir Grinberg

Computer and Information Science Ben-Gurion University of the Negev Beer-Sheva, Israel

Legal NLPLLM PersonalizationLLM Fine-TuningPersonalized Language ModelsLanguage ModelingLegal Reasoning+1
AskPDF
Apr 20, 2026

CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora

Transpilation, or code translation, aims to convert source code from one programming language (PL) to another. It is beneficial for many downstream applications, from modernizing large legacy codebases to augmenting data for low-resource PLs. Recent large language model (LLM)-based approaches have demonstrated immense potential for code translation. Among these approaches, training-based methods are particularly important because LLMs currently do not effectively adapt to domain-specific settings that suffer from a lack of knowledge without targeted training. This limitation is evident in transpilation tasks involving low-resource PLs. However, existing training-based approaches rely on a pairwise transpilation paradigm, making it impractical to support a diverse range of PLs. This limitation is particularly prominent for low-resource PLs due to a scarcity of training data. Furthermore, these methods suffer from suboptimal reinforcement learning (RL) reward formulations. To address these limitations, we propose CodePivot, a training framework that leverages Python as an intermediate representation (IR), augmented by a novel RL reward mechanism, Aggressive-Partial-Functional reward, to bootstrap the model's multilingual transpilation ability without requiring parallel corpora. Experiments involving 10 PLs show that the resulting 7B model, trained on Python-to-Others tasks, consistently improves performance across both general and low-resource PL-related transpilation tasks. It outperforms substantially larger mainstream models with hundreds of billions more parameters, such as Deepseek-R1 and Qwen3-235B-A22B-Instruct-2507, on Python-to-Others tasks and Others-to-All tasks, respectively. In addition, it outperforms its counterpart trained directly on Any-to-Any tasks on general transpilation tasks. The code and data are available at https://github.com/lishangyu-hkust/CodePivot.

Shangyu Li, Juyong Jiang, Meibo Ren +7

The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · Monash University +1

Code TranslationRL for Language ModelsLLM Fine-TuningCode Language Models
AskPDF
Apr 20, 2026

SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models

Safety alignment in large language models is remarkably shallow: it is concentrated in the first few output tokens and reversible by fine-tuning on as few as 100 adversarial examples. This fragility becomes critical in real-world deployment, where models undergo sequential adaptation across domains such as medicine, law, and code, causing safety guardrails to erode cumulatively. Yet all existing safety-preserving methods target only single-task fine-tuning, leaving the multi-domain sequential setting entirely unaddressed. We introduce SafeAnchor, a framework that anchors safety in place throughout continual adaptation. SafeAnchor first identifies low-rank safety subspaces in LoRA parameter space via Fisher Information eigendecomposition, then constrains domain-specific gradient updates to the orthogonal complement of these subspaces, and finally monitors for residual safety drift with threshold-triggered corrective replay. Evaluated on Llama-2-7B-Chat and Mistral-7B-Instruct across a three-domain pipeline and eight benchmarks, SafeAnchor retains 93.2% of original safety alignment, outperforming all baselines by 18-42 points, while matching unconstrained fine-tuning to within 1.5 points on domain tasks.

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong, Hong Kong, China · Brain Investing Limited, Hong Kong, China

Continual Learning for LLMsLLM Fine-TuningLLM Safety AlignmentDomain Adaptation for LLMsGradient ProjectionSafety-Preserving Fine-Tuning of Language Models
AskPDF
Apr 19, 2026

Evaluating Hallucinations in Domain-Adapted Large Language Models

This study investigates the phenomenon of hallucinations in domain-adapted Large Language Models (LLMs), focusing on the fine-tuning of the Llama-2 model with the Lamini dataset. Hallucinations, or the generation of nonsensical or unfaithful content by LLMs, pose a significant challenge, especially when these models are fine-tuned with domain-specific data. Our methodology involves a series of experiments testing memorization, recall, and reasoning capabilities of the fine-tuned LLM, comparing its performance on novel question-answer pairs and domain-specific information. We found that while the model shows proficiency in tasks similar to its training data, its capability to accurately reason about and recall new domain-specific information remains limited, leading to instances of hallucination. The model demonstrates a tendency to provide correct answers with extra information, suggesting an inclination toward over-generation. These results suggest important limitations of fine-tuning-only approaches for mitigating hallucinations when adapting LLMs to specialized domains and underscore the need for more robust methods in adapting LLMs to specialized domains. The study also provides insights into the varying performance of LLMs on different types of information, revealing a comparative weakness in handling domain-specific queries.

Sanchita Porwal, Sai Prasath S, Xingjian Bi +1

College of Computing Georgia Institute of Technology Atlanta, GA 30318, USA

LLM EvaluationDomain AdaptationLLM Fine-TuningDomain Adaptation for LLMs
AskPDF
Apr 19, 2026

Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

Data curation is a critical yet under-explored area in large language model (LLM) training. Existing methods, such as data selection and mixing, operate in an offline paradigm, detaching themselves from training. This separation introduces engineering overhead and makes the curation brittle: the entire pipeline must be re-run under model/task shifts. Moreover, offline methods alter data size through hard filtering or resampling, often sacrificing data diversity and harming generalization. We propose to rethink data curation as an online reweighting problem, where sample importance is dynamically adjusted during training via loss weighting rather than static pre-processing. Specifically, we introduce ADAPT (Adaptive Data reweighting for Pretraining and FineTuning), a dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity-based quality signals, without changing the number of training samples. Unlike offline methods that enforce a static data distribution, ADAPT acts as an implicit curriculum learner, progressively shifting focus from coarse-grained patterns to fine-grained semantic distinctions as the model evolves. Experiments on both instruction tuning and large-scale pretraining show that ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross-benchmark generalization under equal FLOPs.

Wanru Zhao, Yihong Chen, Yuzhi Tang +6

University of Cambridge · OATML, University of Oxford · University of Toronto +2

Language Model PretrainingLLM Fine-TuningTraining Data Curation
AskPDF
Apr 19, 2026

REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations

Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators into its parameters. We propose REALM, which jointly learns the model parameters and a scalar expertise value for each annotator, entirely unsupervised and requiring nothing beyond annotator identity. The key idea is to model each observed label as a mixture between the model's prediction and a uniform random guess, weighted by the annotator's learned expertise. REALM applies to any task with a fixed label set, and extends to multiple tasks via a learned expertise matrix. On four text-classification datasets with real crowdsourced annotations, REALM is the best method in all 1212 configurations of the three heterogeneous-annotator datasets, improving on the strongest applicable baseline, including majority vote and Dawid--Skene aggregation, by +2.9+2.9 points on average. On five question answering benchmarks with simulated noisy labels, it outperforms naive noisy fine-tuning in 152152 of 162162 configurations, by +5.0+5.0 points on average, with gains that grow with model capacity. The learned expertise additionally recovers annotator reliability without ever observing it. Our code is available at https://github.com/sajjad-ucsb/REALM

Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh +1

Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara, CA 93106, USA

Supervised Fine-TuningLLM Fine-TuningNoisy-Label Learning
AskPDF
Apr 19, 2026

Continual Safety Alignment via Gradient-Based Sample Selection

Large language models require continuous adaptation to new tasks while preserving safety alignment. However, fine-tuning on even benign data often compromises safety behaviors, including refusal of harmful requests, truthfulness, and commonsense reasoning. We investigate which training samples cause alignment drift through a data-centric lens. Our empirical analysis shows samples contribute unequally: high-gradient samples cause greater safety degradation and drive models toward pretrained distributions, while moderate-gradient samples enable task learning with minimal alignment loss. We propose gradient-based sample selection that filters high-gradient samples during fine-tuning. Across multiple model families on continual domain tasks, our method substantially improves alignment preservation while maintaining competitive task performance, without requiring curated safe data or architectural modifications. Our method is robust across selection ratios, task orderings, and diverse attack benchmarks.

Thong Bach, Dung Nguyen, Thao Minh Le +1

Applied Artificial Intelligence Initiative (A2I2), Deakin University · Pennsylvania State University

Continual Learning for LLMsLLM Fine-TuningLLM Safety AlignmentTraining Data SelectionSelective Fine-TuningSafety-Preserving Fine-Tuning of Language Models
AskPDF
Apr 19, 2026

Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition

Extracting clinically relevant information from unstructured medical narratives such as admission notes, discharge summaries, and emergency case histories remains a challenge in clinical natural language processing (NLP). Medical Entity Recognition (MER) identifies meaningful concepts embedded in these records. Recent advancements in large language models (LLMs) have shown competitive MER performance; however, evaluations often focus on general entity types, offering limited utility for real-world clinical needs requiring finer-grained extraction. To address this gap, we rigorously evaluated the open-source LLaMA3 model for fine-grained medical entity recognition across 18 clinically detailed categories. To optimize performance, we employed three learning paradigms: zero-shot, few-shot, and fine-tuning with Low-Rank Adaptation (LoRA). To further enhance few-shot learning, we introduced two example selection methods based on token- and sentence-level embedding similarity, utilizing a pre-trained BioBERT model. Unlike prior work assessing zero-shot and few-shot performance on proprietary models (e.g., GPT-4) or fine-tuning different architectures, we ensured methodological consistency by applying all strategies to a unified LLaMA3 backbone, enabling fair comparison across learning settings. Our results showed that fine-tuned LLaMA3 surpasses zero-shot and few-shot approaches by 63.11% and 35.63%, respectivel respectively, achieving an F1 score of 81.24% in granular medical entity extraction.

Nwe Ni Win, Jim Basilakis, Steven Thomas +6

Western Sydney University, Sydney, Australia · Western Sydney University, Australia · South Western Emergency Research Institute, Sydney, Australia +7

Named Entity RecognitionHealthcareLLM Fine-TuningFew-Shot LearningClinical Information ExtractionLLM Information Extraction
AskPDF
Apr 19, 2026

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment

Fine-tuning well-aligned large language models (LLMs) on new domains often degrades their safety alignment, even when using benign datasets. Existing safety alignment techniques primarily focus on pretraining, leaving fine-tuned models vulnerable to behavioral shifts. In this work, we introduce safety token regularization (STR), a lightweight method designed to preserve safety properties during fine-tuning. Our approach identifies salient tokens from rejection templates of well-aligned models and constrains their associated logits during training, preventing the loss of critical safety behaviors. Unlike reinforcement learning or preference optimization methods, STR requires minimal additional computation and seamlessly integrates with parameter-efficient fine-tuning techniques such as LoRA. Comprehensive experiments demonstrate that our approach achieves safety performance on par with state-of-the-art methods, while preserving task-specific utility and requiring minimal implementation overhead. Furthermore, we show that safety token regularization enhances training stability and overall performance beyond safety considerations alone. This work offers a practical and readily deployable strategy for continual safety alignment in fine-tuned LLMs.

Thong Bach, Truyen Tran

Applied Artificial Intelligence Initiative (A2I2), Deakin University

LLM Fine-TuningLLM Safety Alignment
AskPDF
Apr 18, 2026

How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. We investigate how tokenization influences text-only LMs' ability to represent phonological knowledge. Through a series of probing experiments, we show that subword-based tokenization systematically weakens the encoding of both local (e.g., rhyme) and global (e.g., syllabification) phonological features. To quantify this effect, we introduce the syllabification-tokenization alignment distance (STAD), a metric that measures the misalignment between a model's tokenization and the natural syllable boundaries of words, and find that higher misalignment correlates with poorer phonological representations, providing a simple diagnostic for phonology-aware tokenization. To address these limitations, we propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs, leading to consistent improvements across three phonology-related tasks while largely preserving math and general reasoning ability, with 1.1% and 0.9% drops on GSM8K and MMLU, respectively.

Disen Liao, Freda Shi

University of Waterloo · Vector Institute

LLM Fine-TuningLanguage ModelingSubword TokenizationTokenization in Language ModelsTokenization
AskPDF
Apr 18, 2026

Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

Large Language Models (LLMs) have demonstrated excellent performance in general language understanding, generation and other tasks. However, when fine-tuning for specific domain tasks, the general knowledge accumulated in the pre-training phase is often partially overwritten or forgotten due to parameter updates, which severely limits the generalization ability and transferability of LLMs. Traditional fine-tuning strategies mostly train on the entire parameter space, ignoring the heterogeneity of model parameters, that is, some parameters are extremely important for general tasks, while other parameters are more sensitive to specific tasks. To alleviate the above problems, this paper innovatively proposes a parameter element importance evaluation method, which divides parameters into "core parameters" and "non-core parameters" by distinguishing the importance of parameters for general language ability tasks and specific domain tasks, and fixes the core parameters during fine-tuning, and only fine-tunes the non-core parameters. Extensive experiments on scientific, medical and physical tasks using GPT-J and LLaMA-3 show that our method can mitigate catastrophic forgetting while enhancing the adaptability of the model.

Weijie Wan, Jiangjiang Zhao

Shortest Path Technology · China Mobile Communications Group Co.,Ltd

Continual Learning for LLMsLLM Fine-TuningDomain Adaptation for LLMsSelective Fine-TuningParameter-Efficient Fine-Tuning
AskPDF
Apr 18, 2026

Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models

Recent deployments of large language models (LLMs) as autonomous trading agents raise questions about whether financial decision-making competence generalizes beyond specific market patterns and how it should be trained and evaluated in noisy markets lacking ground truth. We propose a structured framework for training and evaluating such models. Central to our approach is a curated, multiple-choice question (MCQ) dataset derived from classic textbooks and historical markets, verified by an AI committee, enriched with structured reasoning traces, and augmented to reduce shortcut learning. To evaluate whether performance on isolated MCQs generalizes to real-world trading, we introduce a two-stage protocol combining test-set evaluation with an MCQ-based chronological trading simulation. Extensive evaluations across market regimes provide statistically robust evidence that open models trained with our framework exhibit competitive, risk-aware behavior over time, outperform open-source baselines, and approach frontier-model performance at smaller scale. We release the dataset and evaluation framework to support further research.

Yuchen Pan, Soung Chang Liew

Department of Information Engineering, The Chinese University of Hong Kong

LLM EvaluationFinancial Reasoning in Language ModelsLLM Fine-TuningFinancial Reasoning BenchmarksFinanceAlgorithmic Trading
AskPDF
Apr 17, 2026

Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs

Fine-tuning on benign data is known to degrade safety alignment in text and vision LLMs, but whether distinct input properties drive this vulnerability differently remains unclear. Audio introduces a richer problem where benign samples can neighbor harmful content through what is said or how it sounds. We present the first systematic study of benign fine-tuning safety in Audio LLMs, evaluating three state-of-the-art models with a proximity-based framework that decomposes embedding-space distance into semantic, acoustic, and mixed axes. We find that the dominant vulnerability axis is architecture-conditioned, determined by how each model's encoder and projector transform audio into the backbone LLM's input space. Across three models, benign fine-tuning elevates Jailbreak Success Rate (JSR) from single digits to as high as 87%, with the most damaging axis shifting from semantic to acoustic proximity depending on encoder design. Mechanistically, fine-tuning selectively suppresses late-layer refusal circuits while frozen encoders preserve upstream representations: the model still detects harmful content but stops refusing, a recognition-refusal dissociation. Two practical defenses, filtering training data to maximize distance from harmful embeddings and a textual system prompt at inference, reduce JSR to near-zero without architectural modification. These findings show that safety evaluation should account for modality and architecture, while highlighting Audio LLMs as a useful testbed for understanding alignment fragility.

Jaechul Roh, Virat Shejwalkar, Amir Houmansadr

University of Massachusetts Amherst

Language Model Safety EvaluationLLM Fine-TuningLLM Safety AlignmentAudio-Language ModelsLLM Safety
AskPDF
Apr 16, 2026

LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance

Existing research on large language models (LLMs) for automated code compliance has primarily focused on performance, treating the models as black boxes and overlooking how training decisions affect their interpretive behavior. This paper addresses this gap by employing a perturbation-based attribution analysis to compare the interpretive behaviors of LLMs across different fine-tuning strategies such as full fine-tuning (FFT), low-rank adaptation (LoRA) and quantized LoRA fine-tuning, as well as the impact of model scales which include varying LLM parameter sizes. Our results show that FFT produces attribution patterns that are statistically different and more focused than those from parameter-efficient fine-tuning methods. Furthermore, we found that as model scale increases, LLMs develop specific interpretive strategies such as prioritizing numerical constraints and rule identifiers in the building text, albeit with performance gains in semantic similarity of the generated and reference computer-processable rules plateauing for models larger than 7B. This paper provides crucial insights into the explainability of these models, taking a step toward building more transparent LLMs for critical, regulation-based tasks in the Architecture, Engineering, and Construction industry.

Jack Wei Lun Shi, Minghao Dang, Wawan Solihin +1

Department of Civil and Environmental Engineering, National University of Singapore, 117576, Singapore · School of Transportation Science and Engineering, Harbin Institute of Technology, Harbin 150090, China · Research and Innovation, NovaCITYNETS Pte.Ltd., 348615, Singapore

Feature AttributionLLM InterpretabilityLLM Fine-Tuning
AskPDF
Apr 16, 2026

Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch

Recent developments in reasoning capabilities have enabled large language models to solve increasingly complex mathematical, symbolic, and logical tasks. Interestingly, while reasoning models are often trained to generate monolingual text, these models have also been observed to code-switch (i.e., mix languages). Prior works have either viewed code-switching as an undesirable error, attempted to control code-switching through modifications to input prompts or the output decoding process, or focus on narrow subsets of languages, domains, tasks, and models. We address these gaps by introducing the first linguistically and behaviorally motivated fine-tuning framework for identifying beneficial code-switched reasoning behaviors in large language models and teaching these models to code-switch more effectively for reasoning. First, we create and systematically analyze a dataset of reasoning traces from diverse models, languages, tasks, and domains to understand the types of code-switching behaviors found in existing reasoning models. Then, we develop fine-tuning interventions that teach reasoning models to code-switch based on our observations of helpful behaviors in existing models. We find that our framework can significantly increase beneficial code-switched reasoning behaviors in a data-efficient manner. Interestingly, we also find that code-switching behaviors in reasoning models can be modified by fine-tuning for tasks that do not directly demonstrate code-switching in reasoning (e.g., machine translation). Our work suggests that data-efficient interventions can instill helpful forms of code-switching behavior in reasoning models.

Eleanor M. Lin, David Jurgens

University of Michigan

Multilingual Language ModelsCode-SwitchingCross-Lingual Reasoning in Language ModelsLLM Fine-TuningLarge Reasoning Models
AskPDF
Apr 16, 2026

Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation

Recent advances in large language models have improved code generation, but their use in hardware description languages is still limited. Moreover, training data and testbenches for these models are often scarce. This paper presents a workflow that uses multi-agent models to generate testbenches for high-quality fine-tuning data. By automating testbench creation, the fine-tuned model for the specification-to-Verilog task achieves performance comparable to state-of-the-art methods on the refined VerilogEval v2 benchmark while using less training data. This study provides a basis for future work on LLM-based HDL generation and automated verification.

Mu-Chi Chen, Po-Hsuan Huang, Yu-Hung Kao +6

Academia Sinica, Taipei, Taiwan · National Taiwan University, Taipei, Taiwan · National Cheng Kung University, Tainan, Taiwan

LLM Fine-TuningCode Generation
AskPDF
Apr 16, 2026

Fact4ac at the Financial Misinformation Detection Challenge Task: Reference-Free Financial Misinformation Detection via Fine-Tuning and Few-Shot Prompting of Large Language Models

The proliferation of financial misinformation poses a severe threat to market stability and investor trust, misleading market behavior and creating critical information asymmetry. Detecting such misleading narratives is inherently challenging, particularly in real-world scenarios where external evidence or supplementary references for cross-verification are strictly unavailable. This paper presents our winning methodology for the "Reference-Free Financial Misinformation Detection" shared task. Built upon the recently proposed RFC-BENCH framework (Jiang et al. 2026), this task challenges models to determine the veracity of financial claims by relying solely on internal semantic understanding and contextual consistency, rather than external fact-checking. To address this formidable evaluation setup, we propose a comprehensive framework that capitalizes on the reasoning capabilities of state-of-the-art Large Language Models (LLMs). Our approach systematically integrates in-context learning, specifically zero-shot and few-shot prompting strategies, with Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA) to optimally align the models with the subtle linguistic cues of financial manipulation. Our proposed system demonstrated superior efficacy, successfully securing the first-place ranking on both official leaderboards. Specifically, we achieved an accuracy of 95.4% on the public test set and 96.3% on the private test set, highlighting the robustness of our method and contributing to the acceleration of context-aware misinformation detection in financial Natural Language Processing. Our models (14B and 32B) are available at https://huggingface.co/KaiNKaiho.

Cuong Hoang, Le-Minh Nguyen

Japan Advanced Institute of Science and Technology

LLM Fine-TuningAutomated Fact-CheckingFinance
AskPDF
Apr 15, 2026

TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

While Large Language Models (LLMs) have empowered AI research agents to perform isolated scientific tasks, automating complex, real-world workflows, such as LLM training, remains a significant challenge. In this paper, we introduce TREX, a multi-agent system that automates the entire LLM training life-cycle. By orchestrating collaboration between two core modules-the Researcher and the Executor-the system seamlessly performs requirement analysis, open-domain literature and data research, formulation of training strategies, preparation of data recipes, and model training and evaluation. The multi-round experimental process is modeled as a search tree, enabling the system to efficiently plan exploration paths, reuse historical results, and distill high-level insights from iterative trials. To evaluate the capability of automated LLM training, we construct FT-Bench, a benchmark comprising 10 tasks derived from real-world scenarios, ranging from optimizing fundamental model capabilities to enhancing performance on domain-specific tasks. Experimental results demonstrate that the TREX agent consistently optimizes model performance on target tasks.

Zerun Ma, Guoqiang Wang, Xinchen Xie +7

Shanghai AI Laboratory · Fudan University

LLM Fine-TuningLLM Agent TrainingMulti-Agent LLM Systems
AskPDF
Apr 15, 2026

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.

Yarui Cao, Kai Liu

Department of Computer Science, Clemson University, Clemson, SC 29634, USA

LLM Fine-TuningParameter-Efficient Fine-TuningLow-Rank Adaptation
AskPDF
Apr 14, 2026

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.

Shanyong Wang, Shuhang Lin, Yining Zhao +2

Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign

LLM Fine-TuningPersonalized Language Model AlignmentPreference AlignmentValue AlignmentUser Preference ModelingMulti-Objective Language Model Alignment
AskPDF
Apr 14, 2026

Analyzing the Effect of Noise in LLM Fine-tuning

Fine-tuning is the dominant paradigm for adapting pretrained large language models (LLMs) to downstream NLP tasks. In practice, fine-tuning datasets may contain various forms of noise arising from annotation errors, preprocessing artifacts, or automated data collection. While prior work has focused on designing robust learning algorithms to mitigate performance degradation under noisy conditions, comparatively little is known about how different types of noise affect the internal learning dynamics of LLMs during fine-tuning. In this work, we systematically study the impact of noise on model behavior across three pretrained model families (GPT-2, Qwen2 and Llama-2) and three diverse NLP tasks. We introduce controlled perturbations corresponding to three common real-world noise types: label noise, grammatical noise, and typographical noise. Beyond task-level performance, we analyze layer-wise representation changes and attention patterns to understand how noise propagates through the network. Our results show that corrupting labels (i.e. label noise) consistently causes the largest performance degradation, whereas grammatical noise and typographical noise can occasionally yield mild regularization benefits. We further find that noise effects are localized primarily to task-specific layers, while attention structures remain comparatively stable.

Lingfang Li, Procheta Sen

LLM Fine-TuningLanguage Model RobustnessNoisy-Label Learning
AskPDF
Apr 13, 2026

CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation

With the recent progress of Large Language Models (LLMs), there is a growing interest in applying these models to solve complex and challenging problems. Modern LLMs, capable of processing long contexts and generating verbalized explanations, offer significant potential in addressing real-world applications. However, a critical hurdle in deploying LLMs for practical decision-making is their inability to provide reliable, quantitative probabilities. While task-specific fine-tuning of LLMs using traditional discriminative objectives (similar to encoder-only models) can yield probability estimates, this often leads to catastrophic forgetting and linguistic collapse. Consequently, the model loses its ability to generate explanations, severely undermining its interpretability and usability. To address this challenge, we propose CLSGen, a novel LLM fine-tuning framework designed for binary classification tasks. The CLSGen framework encompasses a new model architecture, training methodology, and data construction strategy to enable robust probability estimation without sacrificing the model's inherent explanation-generation capabilities. Experimental results across multiple benchmark datasets demonstrate that models fine-tuned with CLSGen outperform existing baselines in classification metrics (AUROC and F1-score). Regarding explanation, the results showed strong alignment between predicted labels and generated justifications, as well as high readability.

WonJin Yoon, Kangyu Zhu, Ian Bulovic +5

Boston Children’s Hospital · Harvard Medical School · University of Colorado Anschutz Medical Campus +2

LLM Fine-TuningConfidence Estimation in Language ModelsFaithfulness of Language Model Explanations
AskPDF
Apr 13, 2026

Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuning

In recent years, rapid advances in Multimodal Large Language Models (MLLMs) have increasingly stimulated research on ancient Chinese scripts. As the evolution of written characters constitutes a fundamental pathway for understanding cultural transformation and historical continuity, how MLLMs can be systematically leveraged to support and advance text evolution analysis remains an open and largely underexplored problem. To bridge this gap, we construct a comprehensive benchmark comprising 11 tasks and over 130,000 instances, specifically designed to evaluate the capability of MLLMs in analyzing the evolution of ancient Chinese scripts. We conduct extensive evaluations across multiple widely used MLLMs and observe that, while existing models demonstrate a limited ability in glyph-level comparison, their performance on core tasks-such as character recognition and evolutionary reasoning-remains substantially constrained. Motivated by these findings, we propose a glyph-driven fine-tuning framework (GEVO) that explicitly encourages models to capture evolutionary consistency in glyph transformations and enhances their understanding of text evolution. Experimental results show that even models at the 2B scale achieve consistent and comprehensive performance improvements across all evaluated tasks. To facilitate future research, we publicly release both the benchmark and the trained models\footnote{https://github.com/songruiecho/GEVO}.

Rui Song, Lida Shi, Ruihua Qi +2

College of Computer Science and Technology, Jilin University, China · Key Laboratory of Ancient Chinese Script, Culture Relics and Artificial Intelligence, Jilin University, China · School of Artificial Intelligence, Jilin University, China +1

Multimodal Large Language ModelsLLM Fine-TuningMultimodal Model Evaluation
AskPDF
Apr 12, 2026

Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series

The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas +2

SpeakLeash · Azurro · ACK Cyfronet AGH +2

Language ModelingSubword TokenizationLanguage Model PretrainingLLM Fine-Tuning
AskPDF
Apr 12, 2026

Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions

Low-rank adaptation (LoRA) is a widely used strategy for efficient fine-tuning of large language models (LLMs), but its strictly linear structure fundamentally limits expressive capacity. The bilinear formulation of weight updates captures only first-order dependencies between low-rank factors, restricting the modeling of nonlinear and higher-order parameter interactions. In this paper, we propose Polynomial Expansion Rank Adaptation (PERA), a novel method that introduces structured polynomial expansion directly into the low-rank factor space. By expanding each low-rank factor to synthesize high-order interaction terms before composition, PERA transforms the adaptation space into a polynomial manifold capable of modeling richer nonlinear coupling without increasing rank or inference cost. We provide theoretical analysis demonstrating that PERA offers enhanced expressive capacity and more effective feature utilization compare to existing linear adaptation approaches. Empirically, PERA consistently outperforms state-of-the-art methods across diverse benchmarks. Notably, our experiments show that incorporating high-order nonlinear components particularly square terms is crucial for enhancing expressive capacity and maintaining strong and robust performance under various rank settings. Our code is available at https://github.com/zhangwenhao6/PERA

Wenhao Zhang, Lin Mu, Li Ni +2

Anhui University · University of Science and Technology of China

Nonlinear LoRALLM Fine-TuningLow-Rank AdaptationParameter-Efficient Fine-Tuning
AskPDF
Apr 11, 2026

Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models

Supervised Fine-Tuning (SFT) is the standard approach for adapting large language models (LLMs) to downstream tasks. However, we observe a persistent failure mode: even after convergence, models often fail to correctly reproduce a subset of their own supervised training data. We refer to this behavior as the Incomplete Learning Phenomenon(ILP). This paper presents the first systematic study of ILP in LLM fine-tuning. We formalize ILP as post-training failure to internalize supervised instances and demonstrate its prevalence across multiple model families, domains, and datasets. Through controlled analyses, we identify five recurrent sources of incomplete learning: (1) missing prerequisite knowledge in the pre-trained model, (2) conflicts between SFT supervision and pre-training knowledge, (3) internal inconsistencies within SFT data, (4) left-side forgetting during sequential fine-tuning, and (5) insufficient optimization for rare or complex patterns. We introduce a diagnostic-first framework that maps unlearned samples to these causes using observable training and inference signals, and study several targeted mitigation strategies as causal interventions. Experiments on Qwen, LLaMA, and OLMo2 show that incomplete learning is widespread and heterogeneous, and that improvements in aggregate metrics can mask persistent unlearned subsets. The findings highlight the need for fine-grained diagnosis of what supervised fine-tuning fails to learn, and why.

Chao Xue, Yao Wang, Mengqiao Liu +11

University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China +2

LLM Fine-TuningSupervised Fine-TuningLLM Training
AskPDF