While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
Figures & tables
Figure 1: Generative Performance vs. Trainable Parameters on ConvAI2. SAPE maintains high semantic fidelity (BERTScore) even at extreme sub-million parameter constraints.
Figure 2: Architectural overview of the SAPE framework. The diagram illustrates the macro-to-micro structural constraints of the proposed sandwich topology. (a) Sandwich Topology: Network-level parameter sharing across N Transformer layers, depicting isolated boundary adapters ( A(0)=θL0,A(N−1)=θLN−1 ) alongside shared intermediate groups ( Ig⟹A(l)=θg ). (b) Intra-Layer Placement: Detailing the injection of the bottleneck module Fθ immediately following the final feed-forward sub-layer, receiving the hidden state hl . (c) Adapter Module: Internal tensor operations of the bottleneck adapter, highlighting the down-projection ( Wdown∈Rd×m ), non-linear ReLU activation σ(⋅) , dropout regularization, and up-projection ( Wup∈Rm×d ).
Figure 3: Epoch-wise validation trajectories for SAPE ( D=16 ) and AdaLoRA across low-resource tasks. To ensure a strict and fair evaluation, AdaLoRA is implemented using the exact hyperparameter configurations and initialization schedules specified in its original paper.
Figure 4: Architectural ablation of the SAPE model comparing bottleneck capacity ( D ) and topological granularity ( G ) across rigid reasoning (GSM8K), semantic reasoning (CSQA), and generative fidelity (ConvAI2).
Topology
Distinct L0
Distinct LN−1
SST-2
CoLA
QNLI
RTE
Avg.
Normal Sharing
No
No
95.18
67.65
49.46
82.31
73.65
Bottom-Slice
Yes
No
94.50
66.85
92.99
82.31
84.16
Top-Slice
No
Yes
95.18
66.80
81.93
82.67
81.65
Sandwich-Style
Yes
Yes
95.18
66.44
92.82
84.48
84.73
Table 3: Ablation study on adapter placement topologies across GLUE benchmark tasks using DeBERTa-base. The Sandwich-Style configuration prevents boundary interference and yields the highest average.
Configuration / Group ( G )
Params
SST-2
CoLA
QNLI
RTE
Avg.
G=1
76K
95.41
68.81
93.63
85.92
85.94
G=2
101K
96.10
68.37
93.79
87.36
86.41
G=4 ( DenseLate )
152K
95.76
69.24
93.68
84.84
85.88
G=4 ( DenseEarly )
152K
96.10
69.27
93.52
85.56
86.11
G=5
178K
96.44
67.31
93.77
85.56
85.77
G=7 ( DenseLate )
228K
95.99
68.28
93.43
85.56
85.82
Table 4: Ablation on intermediate group granularity ( G ) and structural density allocation across GLUE benchmarks.
Strategy / Method
Config
Params
SST-2
CoLA
QNLI
RTE
Avg.
Severe Bottleneck (PAdapter)
G=10,d=5
101K
95.76
66.06
93.59
82.67
84.52
Aggressive Sharing (SAPE)
G=2,d=16
101K
96.10
68.37
93.79
87.36
86.41
Table 5: Comparison of parameter reduction strategies under an equivalent budget ( ∼101K parameters).
Masking Strategy
Dropout ( p )
SST-2
CoLA
QNLI
RTE
Avg.
Shared Mask (across group)
0.10
95.76
66.98
93.52
85.56
85.46
No Dropout
0.00
95.99
66.64
93.33
83.75
84.93
Independent Mask (SAPE)
0.10
96.10
68.37
93.79
87.36
86.41
Table 6: Ablation study on dropout masking mechanics within shared adapter layers.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch Size (Train / Eval)
32
Learning Rate
1×10−4
Optimizer
AdamW *
LR Scheduler
Linear *
Weight Decay
0.0 *
Warmup Ratio
0.0 *
Appendix
Table 10: Hyperparameter and architectural configurations for the SAPE topology ablation study on DeBERTa V3-base . All four routing strategies (Regular, Top-Slice, Bottom-Slice, and Sandwich) were trained under identical optimization conditions to ensure a fair parameter budget. Asterisks (*) denote implicit framework defaults.
Popular low-rank parameter-efficient fine-tuning (PEFT) methods represent adaptation as separate updates to selected backbone weights, without maintaining an explicit task-specific state that is updated and reused across depth. These updates also require the backbone at inference and therefore cannot operate as standalone predictors. We propose ShadowPEFT, which consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. A persistent shadow state refines the frozen backbone representations and is updated from them in an interactive manner. Because the shadow model is trained as a complete predictor, it can be detached for shadow-only inference without executing the base model and can be initialized from a pretrained model. Experiments on text and image generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.
Xianming Li, Zongxi Li, Tsz-fung Andrew Lee +3
PolyU · School of Data Science, Lingnan University, Hong Kong SAR
Parameter-Efficient Fine-Tuning (PEFT) commonly adapts large language models using a single shared Low-Rank Adapter (LoRA). This shared optimization space often suffers from interference when adapting heterogeneous task sequences, leading to poor transfer and catastrophic forgetting. Existing approaches mainly improve adapter expressiveness by increasing parameter capacity or composing multiple adapters, yet they still rely on a shared optimization path. In this paper, we propose an optimization-path organization framework for parameter-efficient fine-tuning of large language models, implemented as an automatic multi-policy PEFT architecture. Specifically, optimization-compatible adaptation paths are automatically organized through task grouping and task sequencing under a fixed parameter budget. The organized optimization paths are implemented as independent Quantized Low-Rank Adapters (QLoRA), enabling heterogeneous tasks to be optimized in decoupled adaptation spaces while preserving positive transfer among compatible tasks. Experiments on the TRACE benchmark demonstrate that performance consistently improves from conventional single-policy PEFT to multi-policy PEFT, with the proposed automatic multi-policy framework achieving the best performance of 44.78 under the same trainable capacity. This suggests that optimization-path organization is more effective than simply increasing adapter capacity for heterogeneous parameter-efficient fine-tuning.
Jiajia Tang, Sizhe Yuen, Francisco Gomez Medina +2
The Alan Turing Institute, London, UK · King’s College London, London, UK · University of Southampton, Southampton, UK
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.