New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework

Shaocong Ma; Peiran Yu; Heng Huang

cs.AIApr 10, 2026

New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework

Authors: Shaocong Ma, Peiran Yu, Heng Huang

Organizations: Department of Computer Science University of Maryland College Park, MD 20742, USA · Department of Computer Science and Engineering University of Texas Arlington Arlington, TX 76019, USA

Abstract

Fine-tuning Large Language Models (LLMs) typically involves either full fine-tuning, which updates all model parameters, or Parameter-Efficient Fine-Tuning (PEFT), which adjusts a small subset of parameters. However, both approaches have inherent limitations: full fine-tuning is computationally expensive, while PEFT often struggles to learn new knowledge and exhibits suboptimal performance. To overcome these issues, we propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth-order and first-order optimization methods. To analyze our new algorithm, we develop a theoretical framework centered on the concept of hybrid smoothness condition, which accounts for the heterogeneous nature of the optimization landscape in joint LLM and PEFT training. We derive a rigorous convergence analysis for the convergence of reshuffling-type SGD algorithm under multiple learning rates and demonstrate its effectiveness through extensive empirical studies across various downstream tasks and model architectures. On the practical side, our results demonstrate consistent performance improvement, making the approach a viable solution for large-scale language model fine-tuning.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of smoothness structures in hybrid fine-tuning a large language model. These complex characteristics pose new challenges for the convergence analysis of traditional optimization algorithms, motivating us to consider a relaxed smoothness condition, hybrid smoothness condition ( Definition 1 ), for the hybrid fine-tuning method.

Panel of Figure 2 — Figure 2: Comparison of training loss curves under different learning rate configurations for full fine-tuning and prompt tuning on the SST-2 dataset ( Socher et al., 2013 ) with the base model OPT-1.3b ( Zhang et al., 2022 ) . This example illustrates the necessaity of using different learning rates in hybrid-tuning structure.

Figure 3: Comparison among Hybrid Fine-Tuning (Hybrid), FO PEFT methods (FO-PEFT), FO full fine-tuning (FO-FT), and ZO full fine-tuning (ZO-FT). In $13/18\approx 72.2\%$ combinations, Hybrid Fine-Tuning outperforms both ZO and FO full fine-tuning.

		Pairwise Comparison							First-Order PEFT vs. Hybrid
Model	Task	SST-2	RTE	WSC	WiC	COPA	WinoG.	Task	SST-2	RTE	WSC	WiC	COPA	WinoG.
Model	Task Type	——— Classification ——–				— Reasoning —		Task Type	——— Classification ——–				— Reasoning —
Llama-2-7b	FO-Prompt	95.6	59.9	36.5	58.5	88.0	67.2	FO-Prompt	95.6	59.9	36.5	58.5	88.0	67.2
	Hybrid-Prompt	95.9	59.9	61.5	64.4	88.0	68.9	FO-Prefix	91.1	60.6	51.9	51.7	83.0	66.2
	FO-Prefix	91.1	60.6	51.9	51.7	83.0	66.2	FO-LoRA	94.6	62.1	60.6	61.6	84.0	68.5
	Hybrid-Prefix	91.6	60.6	42.3	51.5	85.0	64.3	Hybrid-Prompt	95.9	59.9	61.5	64.4	88.0	68.9

Table 1: Experiment results for various fine-tuning methods applied to three large language models across six NLP tasks. Highlighted cells denote the best score for each comparision pair. The left panel (Pairwise Comparison) presents side-by-side comparisons of each Hybrid variant with its corresponding first-order (FO) method ( e.g. FO-Prompt vs. Hybrid-Prompt), enabling direct performance comparisons. The Hybrid method outperforms its FO counterpart in 41 out of 54 cases (

\approx 76\%

). The right panel (First-Order PEFT vs. Hybrid) groups all FO-based methods (Prompt, Prefix, LoRA) separately from their Hybrid counterparts, emphasizing the overall gains from hybrid fine-tuning. In 17 out of 18 comparisons (

\approx 94.5\%

), Hybrid variants yield superior performance.

Figure 4: Training curves for OPT-1.3B model with the prompt tuning on the SST2 dataset. The Hybrid Fine-Tuning method achieves significantly faster convergence than the other two baselines.

Optimizer	Theoretical Memory	Asymptotical Memory	Empirical Memory	Consumed GPU
FO-SGD (LLM)	$\displaystyle\sum_{\ell}\max\!\bigl\{\|a_{\ell}\|,\|x_{\ell}\|\bigr\}+\|x\|$	$\displaystyle\sum_{\ell}\max\!\bigl\{\|a_{\ell}\|,\|x_{\ell}\|\bigr\}$	54 GB	$2\times$ A6000
ZO-SGD (LLM)	$\max_{\ell}\|x_{\ell}\|$	$\max_{\ell}\|x_{\ell}\|$	32 GB	$1\times$ A6000
FO-SGD (Prompt)	$\displaystyle\sum_{\ell}\max\!\bigl\{\|b_{\ell}\|,\|y_{\ell}\|\bigr\}+\|x\|$	$\|x\|$	46 GB	$1\times$ A6000
Hybrid ZO-SGD (LLM+Prompt)	$\displaystyle\sum_{\ell}\max\!\bigl\{\|b_{\ell}\|,\|y_{\ell}\|\bigr\}+\max_{\ell}\|x_{\ell}\|+\|x\|$	$\|x\|$	46 GB	$1\times$ A6000

Table 2: Comparison of theoretical, asymptotical, and empirical memory usage for different fine-tuning optimizers.

|x|

and

|y|

denote the parameter counts of the base LLM and the PEFT module, respectively, with

|y|/|x|\!\to 0

|a|

and

|b|

are the per-layer gradient states kept during optimization.

Panel of Figure 5 — Figure 5: Extended comparison of gradient Lipschitz constant $L$ for OPT-1.3b (Left) and LLaMa-2-7b (Right) in different modules (Base LLM and Prefix Tuning). The base LLM exhibits a significantly larger Lipschitz constant, further confirming our observation in Figure 1(b) .

Table 8

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

		SST2		Copa		WinoGrande
		steps	$\mu$	steps	$\mu$	steps	$\mu$
Llama-2-7b	ZO-FT	$1.1\times 10^{4}$	$10^{-5}$	$1.6\times 10^{4}$	$10^{-4}$	$1.8\times 10^{4}$	$10^{-5}$
	FO-Prompt	$6\times 10^{3}$	/	$9\times 10^{3}$	/	$9\times 10^{3}$	/
	Hybrid-Prompt	$1.5\times 10^{3}$	$10^{-5}$	$5\times 10^{3}$	$10^{-5}$	$9\times 10^{3}$	$10^{-5}$
	FO-Prefix	$2\times 10^{4}$	/	$1.5\times 10^{4}$	/	$3\times 10^{3}$	/
	Hybrid-Prefix	$9.5\times 10^{3}$	$10^{-5}$	$7.5\times 10^{3}$	$10^{-5}$	$9\times 10^{3}$	$10^{-5}$

Appendix

Table 5: A detailed breakdown of the optimal hyperparameters including training steps and

\mu

specified in Eq. 2 and training specifics for each fine-tuning method applied to different model architectures across SST2, Copa, and WinoGrande tasks. Highlighted cells indicate efficient training processes, showcasing the reduced steps required by hybrid approaches to achieve optimal performance.

Panel of Figure 6 — Figure 6: Comparison of training curves for different models and datasets. These results demonstrate that the similar outperformance of hybrid-tuning is observed across various model architectures and NLP tasks.

Figure 7: Training performance of OPT-1.3B on SST-2 using prompt tuning versus wall-clock time. The time cost of each method is: Hybrid Tuning (1.28 sec/step), FO-Prompt (0.80 sec/step), ZO-FT (0.64 sec/step), and one forward pass (0.15 sec/step).

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 250 papers

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Gradient Descent Zeroth-Order Optimization
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University
Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
Mar 14, 2026Haitao Jiang, Wenbo Zhang, Jiarui Yao +3LLM Post-Training Supervised Fine-Tuning
North Carolina State University · UC Irvine · University of Illinois Urbana-Champaign +2
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
May 12, 2026Chengqian Zhang, Wei Zhu, Kyumin LeeLow-Rank Adaptation LLM Post-Training
Worcester Polytechnic Institute · University of Hong Kong

Jun 12, 2026J/K move · Enter open · S save

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin, Aleksandr Beznosikov

1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 3Innopolis University

Gradient Descent Zeroth-Order Optimization LLM Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data. A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation, which requires storing activations, gradients, and optimizer states. Zeroth-order (ZO) optimization offers a memory-efficient alternative, but its performance is highly sensitive to the stepsize and smoothing parameter, often requiring costly task-specific tuning. Parameter-free (PF) optimization addresses this issue by adapting algorithmic parameters without prior knowledge of problem-dependent constants. Moreover, large-scale fine-tuning can benefit from geometry-aware updates that account for the heterogeneous structure of parameter blocks, which can be modeled through methods that exploit linear minimization oracle (LMO). In this work, we study PF adaptation for LMO-based ZO optimization and introduce

\texttt{AdaNAGED}

, a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry. We establish convergence guarantees and validate the method on large-scale LLM fine-tuning task with

\texttt{OPT}-1.3\mathrm{B}

model.

Abstract

Fine-Tuning

Explore similar work

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training