LLM Fine-Tuning

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 83% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 509

Nov 4, 2025cs.CL

AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda

Current large language models excel at broad, general-purpose tasks, but consistently underperform when exposed to highly specialized domains that require deep cultural, linguistic, and subject-matter expertise. In particular, traditional medical systems such as Ayurveda embody centuries of nuanced textual and clinical knowledge that mainstream LLMs fail to accurately interpret or apply. We introduce AyurParam-2.9B, a domain-specialized, bilingual language model fine-tuned from Param-1-2.9B using an extensive, expertly curated Ayurveda dataset spanning classical texts and clinical guidance. AyurParam's dataset incorporates context-aware, reasoning, and objective-style Q&A in both English and Hindi, with rigorous annotation protocols for factual precision and instructional clarity. Benchmarked on BhashaBench-Ayur, AyurParam not only surpasses all open-source instruction-tuned models in its size class (1.5--3B parameters), but also demonstrates competitive or superior performance compared to much larger models. The results from AyurParam highlight the necessity for authentic domain adaptation and high-quality supervision in delivering reliable, culturally congruent AI for specialized medical knowledge.

Sravan Gorugantu, Mohd Nauman, Vijay Devane +7

BharatGen Team

Medical LLMs Healthcare Domain Adaptation for LLMs LLM Fine-Tuning Multilingual Language Models

Oct 28, 2025cs.CL

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

The effectiveness of instruction-tuned Large Language Models (LLMs) is often limited in low-resource linguistic settings due to a lack of high-quality training data. We introduce LuxIT, a novel, monolingual instruction tuning dataset for Luxembourgish developed to mitigate this challenge. We synthesize the dataset from a corpus of native Luxembourgish texts, utilizing DeepSeek-R1-0528, chosen for its shown proficiency in Luxembourgish. Following generation, we apply a quality assurance process, employing an LLM-as-a-judge approach, retaining 227,507 high-quality instruction-answer pairs. To investigate the practical utility of the dataset, we fine-tune 14 smaller-scale LLMs (

\leq

15B parameters) on LuxIT and evaluate them on standardized Luxembourgish proficiency exams and five downstream NLP tasks. Training on LuxIT yields a mean accuracy change of +5.37 percentage points on language exams across all 14 models, with 12 of 14 showing improvement. On NLP downstream tasks, 9 of 14 models improve in macro-averaged F1, though gains on the two benchmarks do not systematically correlate. These results underscore the feasibility of leveraging monolingual synthetic data to improve LLM capabilities in low-resource languages, while highlighting the multi-faceted nature of language proficiency.

Julian Valline, Cedric Lothritz, Siwen Guo +1

Luxembourg Institute of Science and Technology 5 Av. des Hauts-Fourneaux L-4362 Esch-sur-Alzette

LLM Fine-Tuning Synthetic Data Generation Instruction Tuning Synthetic Data Generation for Language Models Low-Resource Language Processing

Oct 7, 2025cs.CR

Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

We propose a system for marking sensitive or copyrighted texts to detect their use in fine-tuning large language models under black-box access with statistical guarantees. Our method builds digital marks'' using invisible Unicode characters organized into (cue'', reply'') pairs. During an audit, prompts containing only cue'' fragments are issued to trigger regurgitation of the corresponding ``reply'', indicating document usage. To control false positives, we compare against held-out counterfactual marks and apply a ranking test, yielding a verifiable bound on the false positive rate. Empirically, we obtain a true positive rate of 96.7% at 0% false positive rate and reply regurgitation rates exceeding 28% per document with only 40 (4%) watermarked documents. The approach is minimally invasive, scalable across many sources, robust to standard processing pipelines, and achieves high detection power even when marked data is a small fraction of the fine-tuning corpus.

Yanming Li, Cédric Eichler, Nicolas Anciaux +3

PETSCRAFT · Petscraft, Inria, Palaiseau, France · INSA CVL, Bourges, France +6

Data Provenance Dataset Watermarking LLM Auditing LLM Fine-Tuning Black-Box Model Auditing

Oct 1, 2025cs.AI

Uncovering the Computational Ingredients of Human-Like Representations in LLMs

The human ability to translate diverse perceptual and linguistic inputs into structured behavior has been thought to rest on learning robust representations of concepts. The rapid advancement of transformer-based large language models (LLMs) has surfaced a diversity of computational ingredients relevant for model building - architectures, fine-tuning methods, and training datasets among others - yet it remains unclear which are most crucial for developing human-like conceptual representations. Further, most current benchmarks are ill-suited to measuring representational alignment, making LLMs' scores on them unreliable for assessing whether they are progressing as cognitive models. We address these limitations by evaluating over 75 models on a triplet similarity task, a method well established in cognitive science for measuring conceptual representations, using concepts from the THINGS database. We find that instruction fine-tuning and larger attention head dimensionality are among the strongest predictors of human alignment, while activation function choice, multimodal pretraining, and parameter size have limited influence on alignment. Correlations between alignment scores and existing benchmark scores reveal that while some benchmarks (e.g., BigBenchHard) better capture representational alignment than others (e.g., MUSR), none fully accounts for the variance in human-model alignment, demonstrating their insufficiency. Taken together, our findings highlight key computational ingredients for advancing LLMs as models of human conceptual representation and address a key gap in LLM evaluation.

Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1

University of Wisconsin–Madison · Stanford University

LLM Evaluation LLM Fine-Tuning Representation Alignment

Sep 29, 2025cs.LG

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Xin Qiu, Yulu Gan, Conor F. Hayes +6

Cognizant AI Lab, San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · University of California, Los Angeles, Los Angeles, CA, USA +1

Fine-Tuning Evolutionary Optimization LLM Fine-Tuning Evolution Strategies RL Fine-Tuning

Sep 26, 2025cs.CL

Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

Distilling reasoning traces from strong teacher models has become the standard recipe for building capable small language models. Yet reasoning traces are 5-20

\times

longer than standard instruction fine-tuning (IFT) outputs, meaning every practitioner who chooses reasoning distillation implicitly forgoes training a larger IFT model on the same compute budget. Whether this trade-off is worthwhile remains unaddressed. We study it with a controlled experiment: a single teacher generates paired IFT and reasoning outputs for identical prompts by toggling only its reasoning mode, isolating supervision format as the sole variable. Training students at five scales (0.5B to 14B) and evaluating on 18 benchmarks, we find that at matched FLOPs, IFT lies on or near the Pareto frontier across the majority of configurations. Reasoning reaches the Pareto frontier only on open-ended tasks at 7B and above. Even there, a sequential curriculum mixing just 25-50% reasoning data with IFT captures most of the accuracy benefit at far lower compute cost.

Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2

1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2

CoT Distillation LLM Fine-Tuning Efficient Language Model Reasoning Efficient Language Model Training Language Model Distillation Teacher-Student Learning

Aug 22, 2025cs.SE

EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention

Code Language Models (CodeLLMs) learn token importance from data correlations, whereas human developers attend selectively to semantically salient code. We present EyeMulator, a model-agnostic method that injects human visual-attention priors into CodeLLM fine-tuning without architectural changes. EyeMulator distills eye-tracking data into semantic salience and gaze-transition priors, then uses them to reweight token-level training losses. Across six backbones, two data regimes, and three CodeXGLUE tasks, the reported configurations yield positive matched-metric deltas in all 36 model-task-setting cells. Effects are largest for structure-preserving completion and translation, while summarization shows smaller but positive METEOR deltas. Session-mode and component-ablation analyses further show that reading, writing, semantic, and transition-derived priors provide complementary signal. Human-attention artifacts are available at https://zenodo.org/records/17205682.

Yifan Zhang, Chen Huang, Yueke Zhang +5

Vanderbilt University1 · National University of Singapore2 · University of Notre Dame3

Code Translation LLM Fine-Tuning Code Language Models

Aug 14, 2025cs.AR

Dataset Construction for Training LLM to Learn Analog Circuit Knowledge

This paper constructs a textual dataset for training large language models (LLMs) to learn analog circuit knowledge and customizes LLM training techniques. For dataset construction, high-quality textbooks are collected and decomposed into fine-grained learning nodes, which are then used to construct structured question-thinking-solution-answer (QTSA) quadruples using a multi-agent framework to capture both final answers and thought processes. The resulting dataset consists of 7.26M tokens of unlabeled data for continual pre-training (CPT) and 112.65M tokens of labeled data for supervised fine-tuning (SFT). We customize the training techniques including initial model selection, training paradigms, regularization techniques, and practical implementation references. Instruct models are identified as suitable training initialization points, an SFT-centric training paradigm is established (finding that CPT provides marginal benefits compared with SFT due to imbalanced data distribution), and SFT with KL divergence regularization can achieve a 2.71 percentage-point improvement over SFT alone. A practical training implementation method is provided for resource-constrained scenarios. Experiments demonstrate that the dataset and training techniques enhance LLMs' analog circuit knowledge. The trained 32B instruct model achieves 84.59% accuracy on the AMSBench-TQA benchmark, showing a 15.67 percentage-point improvement over the initial model. The trained model also shows capability in the operational amplifier design task based on the Atelier framework.

Zihao Chen, Ji Zhuang, Jinyi Shen +11

Fudan University, Shanghai, China

Language Model Pretraining Knowledge Augmentation for Language Models LLM Fine-Tuning

Aug 14, 2025cs.DC

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

To fine-tune large language models (LLMs) over private data, federated learning (FL) has emerged as a promising paradigm. However, the prohibitive memory and communication demands of LLMs render standard FL impractical for resource-constrained edge devices. While split federated learning (SFL) alleviates the computing burdens via model partitioning, existing frameworks still suffer from communication bottlenecks and straggler problem due to the parameter aggregation process. To address these challenges, we propose FlexP-SFT, a novel aggregation-free framework for personalized split federated fine-tuning, which fundamentally eliminates the client-side aggregation process. Crucially, to ensure robust training in the absence of global synchronization, we introduce a layer-flexible alignment strategy to balance personalization and generalization capabilities. We further formulate split-ratio selection as a resource-aware discrete optimization problem that jointly accounts for personalization accuracy and system cost. Our proposed scheme simultaneously enhances personalized performance, reduces communication overhead, and resolves the straggler problem. Extensive results show that FlexP-SFT substantially outperforms baselines in both accuracy and latency, and that the optimized split ratio achieves a better resource-accuracy trade-off than static or memory-only choices.

Jiaxiang Geng, Tianjun Yuan, Pengchao Han +3

Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China +1

Communication-Efficient Distributed Training LLM Fine-Tuning Split Federated Learning Federated Fine-Tuning Federated Learning

Aug 4, 2025cs.LG

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.

Minghao Yan, Zhuang Wang, Zhen Jia +2

University of Wisconsin-Madison · Amazon Web Services

LLM Fine-Tuning Efficient Language Model Training Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Jul 11, 2025cs.CL

Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: A Comparative Study

This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law. In tax consulting practice, clients often describe cases in natural language, making LLMs a prime candidate for supporting automated decision-making and reducing the workload of tax professionals. Given the requirement for legally grounded and well-justified analyses, the propensity of LLMs to hallucinate presents a considerable challenge. The experiments focus on two common methods for enhancing LLM performance: fine-tuning and retrieval-augmented generation (RAG). In this study, these methods are applied on both textbook cases and real-world cases from a tax consulting firm to systematically determine the best configurations of LLM-based systems and assess the legal-reasoning capabilities of LLMs. The findings highlight the potential of using LLMs to support tax consultants by automating routine tasks and providing initial analyses, although current prototypes are not ready for full automation due to the sensitivity of the legal domain. The findings indicate that LLMs, when properly configured, can effectively support tax professionals in VAT tasks and provide legally grounded justifications for decisions. However, limitations remain regarding the handling of implicit client knowledge and context-specific documentation, underscoring the need for future integration of structured background information.

Marina Luketina, Andrea Benkel, Christoph G. Schuetz

University of Applied Sciences Upper Austria – Campus Steyr, Wehrgrabengasse 1–3, Steyr, 4400, Austria · Johannes Kepler University Linz, Altenberger Str. 69, Linz, 4040, Austria

AI-Assisted Decision Making Legal Reasoning LLM Fine-Tuning Retrieval-Augmented Generation

May 28, 2025cs.CL

Learning Composable Chains-of-Thought

A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoning problems, but such annotated data is costly to obtain for every problem of interest. We want reasoning models to generalize beyond their training distribution, and ideally to generalize compositionally: combine atomic reasoning skills to solve harder, unseen reasoning tasks. We take a step towards compositional generalization of reasoning skills when addressing a target compositional task that has no labeled CoT data. We find that simply training models on CoT data of atomic tasks leads to limited generalization, but minimally modifying CoT formats of constituent atomic tasks to be composable can lead to improvements. We can train "atomic CoT" models on the atomic tasks with Composable CoT data and combine them with multitask learning or model merging for better zero-shot performance on the target compositional task. Such a combined model can be further bootstrapped on a small amount of compositional data using rejection sampling fine-tuning (RFT). Results on string operations and natural language skill compositions show that training LLMs on Composable CoT outperforms multitask learning and continued fine-tuning baselines within a given training data budget.

Fangcong Yin, Zeyu Leo Liu, Liu Leqi +2

The University of Texas at Austin · Princeton University

LLM Fine-Tuning CoT Reasoning LLM Training Compositional Generalization LLM Reasoning

May 26, 2025cs.CL

APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation

We present Adjacent Possible Exploration (APE), a selective fine-tuning method for adapting large language models that systematically explores parameter modifications while maintaining model stability. Inspired by evolutionary optimization principles, APE evaluates multiple candidate parameter updates through fine-tuning on small data subsets and accepts only those exceeding a performance threshold. Unlike standard fine-tuning that follows single gradient directions, APE implements a filtered selection process that prevents destabilizing parameter changes while enabling systematic improvement. Our method achieves 33.9% BLEU improvement and 36.2% perplexity reduction on news summarization tasks while using minimal computational resources. The approach provides a practical framework for controlled model adaptation that balances performance gains with representational stability.

Javier Marín

LLM Fine-Tuning Selective Fine-Tuning

May 25, 2025cs.AI

RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data

Large language models (LLMs) are increasingly expected to tackle complex tasks, driven by their expanding applications and users' growing proficiency in crafting sophisticated prompts. However, as the number of explicitly stated requirements increases (particularly more than 10 constraints), LLMs often struggle to accurately follow such complex instructions, which limits their applicability in complex real-world scenarios. To the best of our knowledge, existing datasets do not exceed 10 constraints per instance. To address this challenge, we propose RECAST, an efficient and scalable framework for synthesizing datasets where each example incorporates far more constraints than those in existing benchmarks, aiming to challenge and extend the boundaries of models' ability to follow complex instructions. These constraints are extracted from real-world prompt-response pairs to ensure practical relevance. Using this framework, we construct RECAST-30K, a large-scale, high-quality dataset comprising 30k instances spanning 19 constraint types. Experimental results demonstrate that models finetuned on RECAST-30K substantially improve in following complex instructions while maintaining their general capabilities without degradation. Moreover, RECAST enables automatic verification of constraint satisfaction via rule-based validators for quantitative constraints and LLM-based validators for qualitative ones; the verifiability provided by RECAST enables the design of reward functions for reinforcement learning, which further boosts model performance on complex and challenging tasks.

Zhengkang Guo, Wenhao Liu, Mingchen Xie +13

College of Computer Science and Artificial Intelligence, Fudan University · Xiaohongshu Inc.

LLM Fine-Tuning Synthetic Data Generation Instruction Following Synthetic Data Generation for Language Models

May 22, 2025cs.LG

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

Finetuning open-weight Large Language Models (LLMs) is standard practice for achieving task-specific performance improvements. Until now, finetuning has been regarded as a controlled and secure process in which training on benign datasets leads to predictable behaviors. In this paper, we demonstrate, for the first time, that an adversary can create compromised LLMs that are performant and benign, yet exhibit adversarial behaviors once finetuned by downstream users. To this end, we propose an attack, FAB (Finetuning-activated Adversarial Behaviors), which compromises an LLM via meta-learning techniques that simulate downstream finetuning, explicitly optimizing for the emergence of adversarial behaviors in the finetuned models. At the same time, the compromised LLM is regularized to retain general capabilities and to exhibit no adversarial behaviors prior to finetuning. As a result, when users finetune (e.g., instruction-tuning, distillation, DPO) the seemingly benign model on their own datasets, they unknowingly trigger its dormant adversarial behavior. We experimentally demonstrate the effectiveness of FAB across multiple LLMs and three commonly considered target behaviors: unsolicited advertising, jailbreakability, and over-refusal. We show that FAB-triggers are robust to various finetuning choices made by the user (e.g., dataset, number of steps, scheduler, post-training algorithm). Our findings challenge prevailing assumptions on the security of finetuning, revealing a critical attack vector.

Thibaud Gloaguen, Mark Vero, Robin Staab +1

ETH Zurich

LLM Backdoor Attacks Backdoor Attacks LLM Fine-Tuning

May 10, 2025cs.CV

Batch Augmentation with Unimodal Fine-tuning for Multimodal Fusion of Large Language Models

In this paper, we propose batch augmentation with unimodal fine-tuning for multimodal learning. We start with pre-trained unimodal models. We fine-tune the unimodal models with the application data. After that, we form a Multi-Layer Perceptron (MLP) head that takes information from unimodal models and provides output. Finally, we train the MLP layer and unimodal parts with batch augmentation. Depending on the data, some unimodal models can be replaced by hard-coded scripts or AI agents. The unimodal training can also follow batch augmentation when the data is augmentable. We write a multimodal batch augmentation dataloader script that implements the batch augmentation for the multimodal data. We investigate the proposed method on the FPU23 ultrasound and UPMC Food-101 multimodal datasets. The multimodal large language model (LLM) with the proposed training achieves the best average result among the investigated methods across both datasets. According to our literature search, the proposed method achieves state-of-the-art (SOTA) accuracy of 93.29% on the UPMC Food-101 dataset, while we apply the ViT-L/16 model for vision and the GPT-2 model for text. We share the scripts of the proposed method with traditional counterparts at the following repository: github.com/dipuk0506/multimodal

H M Dipu Kabir, Subrota Kumar Mondal, Mohammad Ali Moni

AI and Cyber Futures Institute, Charles Sturt University, Australia · Rural Health Research Institute, Charles Sturt University, Australia · School of Computer Science and Engineering, Macau University of Science and Technology, Macao

Multimodal Fusion Multimodal Classification Data Augmentation LLM Fine-Tuning

Apr 25, 2025cs.CL

Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Models

Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, and information retrieval. Despite its importance, current NLI systems heavily rely on supervised learning with datasets that often contain annotation artifacts and biases, limiting generalization and real-world applicability. In this work, we apply a reinforcement learning-based approach using Group Relative Policy Optimization (GRPO) for Chain-of-Thought (CoT) learning in NLI, eliminating the need for human-labeled rationales and enabling this type of training on challenging datasets such as ANLI. We fine-tune 7B, 14B, and 32B language models using parameter-efficient techniques (LoRA and QLoRA), demonstrating strong performance across standard and adversarial NLI benchmarks. At the 32B scale, GRPO-trained models generalize better than other supervised baselines in adversarial sets. With AWQ quantization, the 32B model fits within 22GB of CUDA memory. This work provides a scalable and practical framework for building robust NLI systems without sacrificing inference quality.

Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín +1

Department of Computer Systems, Technical University of Madrid, Madrid, Spain · United Arab Emirates University, Al Ain, United Arab Emirates

LLM Fine-Tuning RL for Language Model Reasoning Adversarial Robustness of Language Models Natural Language Inference

Apr 24, 2025cs.LG

Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

Large language models (LMs) offer broad generalization capabilities but require vast amounts of data and computational resources for domain-specific tasks; small models (SMs), in contrast, are more efficient and tailored to specific domains yet lack general-purpose coverage. Taking a collaborative approach, where large and small models work synergistically, can accelerate the adaptation of LLMs to private domains and unlock new potential in AI. This survey presents a comprehensive overview of recent advances and challenges in harnessing the collaborative power of large and small models for private-domain adaptation. It specifically focuses on the unique constraints of cross-boundary environments, where models belong to distinct parties, and examines the resulting tensions among data privacy, model security, integrity, and resource limitations. By analyzing the information flow between distinct model and data stakeholders, we propose a unified taxonomy that classifies research into three primary directions: downward knowledge transfer (LM to SM), upward knowledge transfer (SM to LM), and inference-time collaboration across parties. Drawing on this taxonomy, we analyze the core challenges inherent to cross-boundary information exchange, including data-privacy, model-security, and integrity threats as well as efficiency constraints, and synthesize these into a multi-objective optimization problem that governs practical deployment. Finally, we review key open challenges inherent to such hybrid approaches and outline promising directions for future research. By offering a principled, boundary-centric view of this rapidly evolving landscape, this survey aims to serve as a structured resource for researchers and practitioners advancing privacy-aware, resource-efficient AI deployment.

Yang Liu, Kejia Zhang, Bingjie Yan +11

The Hong Kong Polytechnic University, China · Institute for AI Industry Research, Tsinghua University, China · Institute for AI Industry Research, Tsinghua University, China and Shanghai Jiao Tong University, China +3

Privacy-Preserving Language Models LLM Fine-Tuning LLM Security Multi-LLM Collaboration

Mar 27, 2025cs.CL

Boosting Large Language Models with Mask Fine-Tuning

The large language model (LLM) is typically integrated into the mainstream optimization protocol. However, it remains underexplored whether maintaining the model integrity is \textit{indispensable} for promising performance. In this work, we introduce Mask Fine-Tuning (MFT), a novel LLM fine-tuning paradigm demonstrating that carefully breaking the model's structural integrity can surprisingly improve performance without updating model weights. MFT learns and applies binary masks to well-optimized models, using the standard LLM fine-tuning objective as supervision. Based on fully fine-tuned models, MFT uses the same fine-tuning datasets to achieve consistent performance gains across domains and backbones (e.g., an average gain of 2.70/4.15 on IFEval with LLaMA2-7B/3.1-8B). Detailed ablation studies and analyses examine the proposed MFT from different perspectives, including the sparse ratio and the loss surface. Additionally, when deployed on well-trained models, MFT is compatible with other LLM optimization procedures to improve overall model performance. Furthermore, this study extends the masking operation beyond its conventional use in network pruning for model compression to encompass a broader range of model capabilities.

Mingyuan Zhang, Yue Bai, Huan Wang +4

College of Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University

LLM Fine-Tuning

Mar 1, 2025cs.LG

Distributionally Robust Reinforcement Learning with Human Feedback

Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs). However, existing RLHF methods are non-robust, and their performance deteriorates if the downstream task differs significantly from the preference dataset used in fine-tuning. In order to mitigate this problem, we introduce a distributionally robust RLHF for fine-tuning LLMs. In particular, our goal is to ensure that a fine-tuned model retains its performance even when the distribution of prompts significantly differs from the distribution encountered during fine-tuning. We formulate distributionally robust optimization (DRO) version of two popular fine-tuning methods -- (1) reward-based RLHF and (2) reward-free DPO (direct preference optimization). We propose a minibatch gradient descent based algorithms for both of them, and theoretically prove convergence guarantees for the algorithms. Subsequently, we evaluate our algorithms on an out-of-distribution (OOD) task by first training the model on the Unified-Feedback dataset and evaluating its performance on two different datasets. The experimental results show that our robust training improves the accuracy of the learned reward models on average, and markedly on some tasks, such as reasoning. Furthermore, we show that the robust versions of policy optimization methods, similarly improve performance on OOD tasks.

Debmalya Mandal, Paulius Sasnauskas, Goran Radanovic

Dept. of Computer Science University of Warwick, UK · Department of Computing Science University of Alberta, Edmonton, Canada · Max-Planck Institute for Software Systems, Germany

Direct Preference Optimization LLM Fine-Tuning Distributionally Robust RL Robust RL RL from Human Feedback

Feb 20, 2025cs.CL

LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to each long input. Instead of endlessly extending context windows to fit longer inputs in context, LIFT stores and absorbs the input in parameters. By fine-tuning long inputs into parameters, LIFT enables short-context LLMs to answer questions even when required information is absent from the inference context, avoiding the quadratic input-length complexity of standard long-context models. Rather than simple continued pretraining on new long contexts, LIFT uses carefully designed LLM-generated synthetic tasks to enhance comprehension beyond memorization. To offset fine-tuning overhead, we design a highly optimized pipeline that reduces Time to First Token (TTFT) to under 10 seconds for 8k context. We further analyze LIFT's strengths and limitations, discuss large-scale deployment feasibility, and highlight future research directions. Implementation is open-sourced at https://github.com/MuLabPKU/LIFT.

Yansheng Mao, Yufei Xu, Jiaqi Li +5

Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI

Long-Context QA LLM Fine-Tuning Long-Context Modeling Test-Time Training

Oct 16, 2024cs.CL

Tuning Language Models by Mixture-of-Depths Ensemble

Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the predictive power embedded in late layers. Interpretability tools such as the logit lens show that late-layer representations already carry largely formed, task-relevant predictions; here we ask whether that observation can be turned into an actionable training signal. We find that focusing tuning effort on these layers can yield losses comparable to those of the final layer, with complementary test-time behaviour. Building on this, we introduce a tuning framework, Mixture-of-Depths Ensemble (MoDE), which treats the late layers as an ensemble that contributes to the final logits through learned routing weights. MoDE can be applied on top of any existing tuning method (e.g., LoRA) and, in our experiments, modestly improves reasoning performance at a small parameter overhead. We present MoDE as a mechanism study showing that late-layer logits can be made directly useful for tuning, and that they can substitute for substantially larger trainable modules with comparable performance.

Haoyan Luo, Lucia Specia

University of Cambridge · Imperial College London

LLM Fine-Tuning Language Model Ensembles

Oct 3, 2024cs.LG

Mitigating Memorization In Language Models

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.

Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6

University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3

Memorization in Language Models LLM Fine-Tuning Neural Network Memorization LLM Memory Privacy Leakage in Language Models Memorization in Generative Models+1

May 22, 2024cs.CL

Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. Vikhr'' refers to the name of the Mistral LLM series and means a strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.

Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2

ITMO University · HSE University · Vikhr +2

Language Model Pretraining Multilingual Language Models LLM Fine-Tuning Language Modeling Instruction Tuning Tokenizer Adaptation

Oct 6, 2023cs.LG

Amortizing intractable inference in large language models

Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

Edward J. Hu, Moksh Jain, Eric Elmoznino +4

Mila – Quebec AI Institute, Universit´e de Montr´eal · University of Oxford

Amortized Inference LLM Fine-Tuning CoT Reasoning RL for Language Model Reasoning Generative Flow Networks

Date pendingcs.LG

Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective

Large language models are being widely used across industries to generate text that contributes directly to key performance metrics, such as medication adherence in patient messaging and conversion rates in content generation. Pretrained models, however, often fall short when it comes to aligning with human preferences or optimizing for business objectives. As a result, fine-tuning with good-quality labeled data is essential to guide models to generate content that achieves better results. Controlled experiments, like A/B tests, can provide such data, but they are often expensive and come with significant engineering, logistical, and ethical challenges. Meanwhile, companies have access to a vast amount of historical (observational) data that remains underutilized. In this work, we study the challenges and opportunities of fine-tuning LLMs using observational data. We show that while observational outcomes can provide valuable supervision, directly fine-tuning models on such data can lead them to learn spurious correlations. We present empirical evidence of this issue using various real-world datasets and propose DeconfoundLM, a method that explicitly removes the effect of known confounders from reward signals. In simulation experiments, DeconfoundLM more accurately recovers causal relationships and mitigates failure modes of methods that assume counterfactual invariance, achieving over 16% higher objective score than ODIN and other baselines, when entangled confounding is present. Please refer to the project page for code and related resources.

Erfan Loghmani

Foster School of Business University of Washington Seattle, WA 98195

LLM Alignment LLM Fine-Tuning Causal Inference

Date pendingcs.CL

On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study

Controlling the output of Large Language Models (LLMs) is a central challenge for their reliable deployment, yet a clear understanding of the involved trade-offs remains elusive. Current approaches to conditioning are often evaluated with a narrow focus on their effectiveness at injecting or removing a target concept, neglecting generation quality. We systematically investigate a range of conditioning methods in both injection and removal scenarios. We find that efficient steering methods frequently achieve conditioning at a steep cost to fluency. Furthermore, we identify a critical yet previously overlooked interaction with the training paradigm: activation steering methods are far less effective on instruction-tuned models than on their base counterparts. Simple prompting and full-fledged supervised fine-tuning, on the other hand, are viable options for concept injection, but are not as good at concept removal. Finally, cheaply computed textual metrics highly correlate to costly LLM-as-judge scores, and provide insights on the behavior of conditioning methods.

Iuri Macocco, Pau Rodríguez, Arno Blaas +3

Universitat Pompeu Fabra · Apple · Universitat Pompeu Fabra / ICREA

LLM Fine-Tuning Activation Steering Concept Erasure

Date pendingcs.CL

A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

Most endangered languages lack the parallel data required for machine translation, despite the existence of descriptive grammar books. We introduce a pipeline that uses large language models to extract grammatical rules, example sentences, and lexicons from grammar books and generate synthetic parallel corpora for fine-tuning-rather than feeding grammar content into prompts at inference time, as in prior work. Validated on three typologically diverse low-resource languages-Kalamang (Papuan), Tuatschin (Romance), and Mandan (Siouan)-we show that fine-tuning on synthetic data improves over seed-data baselines in 75% of configurations for Kalamang and 59% for Tuatschin, with best-case ChrF++ gains of +8.8, +5.3, and +3.3 respectively. Through a systematic factorial study across 96 configurations varying target part-of-speech, retrieval granularity, and sample volume, we identify which factor combinations drive gains and where they break down. Our results demonstrate that static linguistic documentation can be repurposed for machine translation fine-tuning, offering a practical path towards translation tools for severely under-resourced languages.

Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger +1

Department of Informatics, University of Zurich · Department of Computational Linguistics, University of Zurich

LLM Fine-Tuning Synthetic Data Generation Low-Resource MT Synthetic Data Generation for Language Models Machine Translation

Date pendingcs.CL

SalamandraTA at WMT 2026 Terminology Shared Task: Hard Examples Are Better Teachers

Terminology-aware translation asks for more than a correct translation: the output must use the exact terms a glossary prescribes. The standard recipe, fine-tuning on glossary-annotated translation pairs, hides an inefficiency: for most examples the glossary prescribes exactly what the model would have produced anyway, so they teach nothing about following a glossary. We therefore keep only the examples where the model's own translation contradicts the glossary. In a controlled study at fixed data volume, this selection alone raises term accuracy from 78.7% to 89.9%. The filtered data, built by a two-way synthetic pipeline on open models, is part of the instruction-tuning mixture of our public release SalamandraTA-7b-instruct v3.0, which, used exactly as released and wrapped in a document-level inference pipeline, forms the BSC submission to the WMT26 Terminology Shared Task Track 1. At the official WMT26 evaluation, our system achieves 94.2% term success at 74.6 chrF++, with only two of the twenty-two submissions outperforming it on both metrics. On last year's benchmark, it also surpasses our GRPO-based system, despite being trained solely with ordinary supervised fine-tuning.

Xixian Liao, Maite Melero

Barcelona Supercomputer Center

Hard Negative Mining LLM Fine-Tuning Machine Translation