APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation

Javier Marín

cs.CLMay 26, 2025

APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation

Authors: Javier Marín

Abstract

We present Adjacent Possible Exploration (APE), a selective fine-tuning method for adapting large language models that systematically explores parameter modifications while maintaining model stability. Inspired by evolutionary optimization principles, APE evaluates multiple candidate parameter updates through fine-tuning on small data subsets and accepts only those exceeding a performance threshold. Unlike standard fine-tuning that follows single gradient directions, APE implements a filtered selection process that prevents destabilizing parameter changes while enabling systematic improvement. Our method achieves 33.9% BLEU improvement and 36.2% perplexity reduction on news summarization tasks while using minimal computational resources. The approach provides a practical framework for controlled model adaptation that balances performance gains with representational stability.

Explore similar work

Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization
Apr 18, 2026Weijie Wan, Jiangjiang ZhaoEfficient Task Adaptation Model Fine-Tuning
Shortest Path Technology · China Mobile Communications Group Co.,Ltd
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Jul 10, 2026Ivan Ilin, Philip Zmushko, Peter RichtárikParameter-Efficient Fine-Tuning Methods Model Fine-Tuning
KAUST
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3Training-Side Stage-Aware Low-Rank Adaptation Large Language Model Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI

Apr 18, 2026, cs.CLJ/K move · Enter open · S save

Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

Weijie Wan, Jiangjiang Zhao

Shortest Path Technology · China Mobile Communications Group Co.,Ltd

Efficient Task Adaptation Model Fine-Tuning Catastrophic Forgetting Importance Natural Language Processing

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated excellent performance in general language understanding, generation and other tasks. However, when fine-tuning for specific domain tasks, the general knowledge accumulated in the pre-training phase is often partially overwritten or forgotten due to parameter updates, which severely limits the generalization ability and transferability of LLMs. Traditional fine-tuning strategies mostly train on the entire parameter space, ignoring the heterogeneity of model parameters, that is, some parameters are extremely important for general tasks, while other parameters are more sensitive to specific tasks. To alleviate the above problems, this paper innovatively proposes a parameter element importance evaluation method, which divides parameters into "core parameters" and "non-core parameters" by distinguishing the importance of parameters for general language ability tasks and specific domain tasks, and fixes the core parameters during fine-tuning, and only fine-tunes the non-core parameters. Extensive experiments on scientific, medical and physical tasks using GPT-J and LLaMA-3 show that our method can mitigate catastrophic forgetting while enhancing the adaptability of the model.

Abstract

Explore similar work

Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation