New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework
Authors: Shaocong Ma, Peiran Yu, Heng Huang
Organizations: Department of Computer Science University of Maryland College Park, MD 20742, USA · Department of Computer Science and Engineering University of Texas Arlington Arlington, TX 76019, USA
Fine-tuning Large Language Models (LLMs) typically involves either full fine-tuning, which updates all model parameters, or Parameter-Efficient Fine-Tuning (PEFT), which adjusts a small subset of parameters. However, both approaches have inherent limitations: full fine-tuning is computationally expensive, while PEFT often struggles to learn new knowledge and exhibits suboptimal performance. To overcome these issues, we propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth-order and first-order optimization methods. To analyze our new algorithm, we develop a theoretical framework centered on the concept of hybrid smoothness condition, which accounts for the heterogeneous nature of the optimization landscape in joint LLM and PEFT training. We derive a rigorous convergence analysis for the convergence of reshuffling-type SGD algorithm under multiple learning rates and demonstrate its effectiveness through extensive empirical studies across various downstream tasks and model architectures. On the practical side, our results demonstrate consistent performance improvement, making the approach a viable solution for large-scale language model fine-tuning.
Figures & tables
Figure 1: Visualization of smoothness structures in hybrid fine-tuning a large language model. These complex characteristics pose new challenges for the convergence analysis of traditional optimization algorithms, motivating us to consider a relaxed smoothness condition, hybrid smoothness condition ( Definition 1 ), for the hybrid fine-tuning method.
Figure 2: Comparison of training loss curves under different learning rate configurations for full fine-tuning and prompt tuning on the SST-2 dataset ( Socher et al., 2013 ) with the base model OPT-1.3b ( Zhang et al., 2022 ) . This example illustrates the necessaity of using different learning rates in hybrid-tuning structure.
Figure 3: Comparison among Hybrid Fine-Tuning (Hybrid), FO PEFT methods (FO-PEFT), FO full fine-tuning (FO-FT), and ZO full fine-tuning (ZO-FT). In 13/18≈72.2% combinations, Hybrid Fine-Tuning outperforms both ZO and FO full fine-tuning.
Pairwise Comparison
First-Order PEFT vs. Hybrid
Model
Task
SST-2
RTE
WSC
WiC
COPA
WinoG.
Task
SST-2
RTE
WSC
WiC
COPA
WinoG.
Task Type
——— Classification ——–
— Reasoning —
Task Type
——— Classification ——–
— Reasoning —
Llama-2-7b
FO-Prompt
95.6
59.9
36.5
58.5
88.0
67.2
FO-Prompt
95.6
59.9
36.5
58.5
88.0
67.2
Hybrid-Prompt
95.9
59.9
61.5
64.4
88.0
68.9
FO-Prefix
91.1
60.6
51.9
51.7
83.0
66.2
FO-Prefix
91.1
60.6
51.9
51.7
83.0
66.2
FO-LoRA
94.6
62.1
60.6
61.6
84.0
68.5
Hybrid-Prefix
91.6
60.6
42.3
51.5
85.0
64.3
Hybrid-Prompt
95.9
59.9
61.5
64.4
88.0
68.9
Table 1: Experiment results for various fine-tuning methods applied to three large language models across six NLP tasks. Highlighted cells denote the best score for each comparision pair. The left panel (Pairwise Comparison) presents side-by-side comparisons of each Hybrid variant with its corresponding first-order (FO) method ( e.g. FO-Prompt vs. Hybrid-Prompt), enabling direct performance comparisons. The Hybrid method outperforms its FO counterpart in 41 out of 54 cases ( ≈76% ). The right panel (First-Order PEFT vs. Hybrid) groups all FO-based methods (Prompt, Prefix, LoRA) separately from their Hybrid counterparts, emphasizing the overall gains from hybrid fine-tuning. In 17 out of 18 comparisons ( ≈94.5% ), Hybrid variants yield superior performance.
Figure 4: Training curves for OPT-1.3B model with the prompt tuning on the SST2 dataset. The Hybrid Fine-Tuning method achieves significantly faster convergence than the other two baselines.
Optimizer
Theoretical Memory
Asymptotical Memory
Empirical Memory
Consumed GPU
FO-SGD (LLM)
ℓ∑max{∣aℓ∣,∣xℓ∣}+∣x∣
ℓ∑max{∣aℓ∣,∣xℓ∣}
54 GB
2× A6000
ZO-SGD (LLM)
maxℓ∣xℓ∣
maxℓ∣xℓ∣
32 GB
1× A6000
FO-SGD (Prompt)
ℓ∑max{∣bℓ∣,∣yℓ∣}+∣x∣
∣x∣
46 GB
1× A6000
Hybrid ZO-SGD (LLM+Prompt)
ℓ∑max{∣bℓ∣,∣yℓ∣}+ℓmax∣xℓ∣+∣x∣
∣x∣
46 GB
1× A6000
Table 2: Comparison of theoretical, asymptotical, and empirical memory usage for different fine-tuning optimizers. ∣x∣ and ∣y∣ denote the parameter counts of the base LLM and the PEFT module, respectively, with ∣y∣/∣x∣→0 . ∣a∣ and ∣b∣ are the per-layer gradient states kept during optimization.
Figure 5: Extended comparison of gradient Lipschitz constant L for OPT-1.3b (Left) and LLaMa-2-7b (Right) in different modules (Base LLM and Prefix Tuning). The base LLM exhibits a significantly larger Lipschitz constant, further confirming our observation in Figure 1(b) .
Table 8
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
SST2
Copa
WinoGrande
steps
μ
steps
μ
steps
μ
Llama-2-7b
ZO-FT
1.1×104
10−5
1.6×104
10−4
1.8×104
10−5
FO-Prompt
6×103
/
9×103
/
9×103
/
Hybrid-Prompt
1.5×103
10−5
5×103
10−5
9×103
10−5
FO-Prefix
2×104
/
1.5×104
/
3×103
/
Hybrid-Prefix
9.5×103
10−5
7.5×103
10−5
9×103
10−5
Appendix
Table 5: A detailed breakdown of the optimal hyperparameters including training steps and μ specified in Eq. 2 and training specifics for each fine-tuning method applied to different model architectures across SST2, Copa, and WinoGrande tasks. Highlighted cells indicate efficient training processes, showcasing the reduced steps required by hybrid approaches to achieve optimal performance.
Figure 6: Comparison of training curves for different models and datasets. These results demonstrate that the similar outperformance of hybrid-tuning is observed across various model architectures and NLP tasks.
Figure 7: Training performance of OPT-1.3B on SST-2 using prompt tuning versus wall-clock time. The time cost of each method is: Hybrid Tuning (1.28 sec/step), FO-Prompt (0.80 sec/step), ZO-FT (0.64 sec/step), and one forward pass (0.15 sec/step).
Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data. A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation, which requires storing activations, gradients, and optimizer states. Zeroth-order (ZO) optimization offers a memory-efficient alternative, but its performance is highly sensitive to the stepsize and smoothing parameter, often requiring costly task-specific tuning. Parameter-free (PF) optimization addresses this issue by adapting algorithmic parameters without prior knowledge of problem-dependent constants. Moreover, large-scale fine-tuning can benefit from geometry-aware updates that account for the heterogeneous structure of parameter blocks, which can be modeled through methods that exploit linear minimization oracle (LMO). In this work, we study PF adaptation for LMO-based ZO optimization and introduce AdaNAGED, a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry. We establish convergence guarantees and validate the method on large-scale LLM fine-tuning task with OPT−1.3B model.
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University
Pre-trained Large Language Model (LLM) exhibits broad capabilities, yet, for specific tasks or domains their attainment of higher accuracy and more reliable reasoning generally depends on post-training through Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). Although often treated as distinct methodologies, recent theoretical and empirical developments demonstrate that SFT and RL are closely connected. This study presents a comprehensive and unified perspective on LLM post-training with SFT and RL. We first provide an in-depth overview of both techniques, examining their objectives, algorithmic structures, and data requirements. We then systematically analyze their interplay, highlighting frameworks that integrate SFT and RL, hybrid training pipelines, and methods that leverage their complementary strengths. Drawing on a representative set of recent application studies from 2023 to 2025, we identify emerging trends, characterize the rapid shift toward hybrid post-training paradigms, and distill key takeaways that clarify when and why each method is most effective. By synthesizing theoretical insights, practical methodologies, and empirical evidence, this study establishes a coherent understanding of SFT and RL within a unified framework and outlines promising directions for future research in scalable, efficient, and generalizable LLM post-training.
Haitao Jiang, Wenbo Zhang, Jiarui Yao +3
North Carolina State University · UC Irvine · University of Illinois Urbana-Champaign +2
Post-training has become essential for adapting large language models (LLMs) to complex downstream behaviors, including instruction following, preference alignment, and multi-step reasoning. Reinforcement learning with verifiable rewards (RLVR) has recently emerged as a particularly effective post-training paradigm for improving reasoning capabilities, with critic-free algorithms such as GRPO and GSPO enabling scalable optimization. However, RLVR post-training with full fine-tuning (FFT) requires substantial GPU memory and incurs high training costs. Although parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), effectively reduce computational costs, they often suffer from a noticeable performance gap compared to full fine-tuning in post-training for complex reasoning tasks. In this paper, we propose Hybrid-LoRA, an efficient hybrid post-training framework that selectively applies full fine-tuning to a small subset of modules less suited to low-rank adaptation, while adapting the remaining components with LoRA. We introduce a novel Hybrid-LoRA Score to rank candidate modules according to their sensitivity to low-rank adaptation under a fixed parameter budget. Experiments show that Hybrid-LoRA closely matches full fine-tuning performance under a 10% full fine-tuning module budget, with the remaining candidate modules adapted by LoRA, consistently outperforming four state-of-the-art PEFT post-training baselines, achieving improvements of up to 5.65% and on average 4.36% over the best baseline.
Chengqian Zhang, Wei Zhu, Kyumin Lee
Worcester Polytechnic Institute · University of Hong Kong