While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
Figures & tables
Figure 1: Generative Performance vs. Trainable Parameters on ConvAI2. SAPE maintains high semantic fidelity (BERTScore) even at extreme sub-million parameter constraints.
Figure 2: Architectural overview of the SAPE framework. The diagram illustrates the macro-to-micro structural constraints of the proposed sandwich topology. (a) Sandwich Topology: Network-level parameter sharing across N Transformer layers, depicting isolated boundary adapters ( A(0)=θL0,A(N−1)=θLN−1 ) alongside shared intermediate groups ( Ig⟹A(l)=θg ). (b) Intra-Layer Placement: Detailing the injection of the bottleneck module Fθ immediately following the final feed-forward sub-layer, receiving the hidden state hl . (c) Adapter Module: Internal tensor operations of the bottleneck adapter, highlighting the down-projection ( Wdown∈Rd×m ), non-linear ReLU activation σ(⋅) , dropout regularization, and up-projection ( Wup∈Rm×d ).
Figure 3: Epoch-wise validation trajectories for SAPE ( D=16 ) and AdaLoRA across low-resource tasks. To ensure a strict and fair evaluation, AdaLoRA is implemented using the exact hyperparameter configurations and initialization schedules specified in its original paper.
Figure 4: Architectural ablation of the SAPE model comparing bottleneck capacity ( D ) and topological granularity ( G ) across rigid reasoning (GSM8K), semantic reasoning (CSQA), and generative fidelity (ConvAI2).
Topology
Distinct L0
Distinct LN−1
SST-2
CoLA
QNLI
RTE
Avg.
Normal Sharing
No
No
95.18
67.65
49.46
82.31
73.65
Bottom-Slice
Yes
No
94.50
66.85
92.99
82.31
84.16
Top-Slice
No
Yes
95.18
66.80
81.93
82.67
81.65
Sandwich-Style
Yes
Yes
95.18
66.44
92.82
84.48
84.73
Table 3: Ablation study on adapter placement topologies across GLUE benchmark tasks using DeBERTa-base. The Sandwich-Style configuration prevents boundary interference and yields the highest average.
Configuration / Group ( G )
Params
SST-2
CoLA
QNLI
RTE
Avg.
G=1
76K
95.41
68.81
93.63
85.92
85.94
G=2
101K
96.10
68.37
93.79
87.36
86.41
G=4 ( DenseLate )
152K
95.76
69.24
93.68
84.84
85.88
G=4 ( DenseEarly )
152K
96.10
69.27
93.52
85.56
86.11
G=5
178K
96.44
67.31
93.77
85.56
85.77
G=7 ( DenseLate )
228K
95.99
68.28
93.43
85.56
85.82
Table 4: Ablation on intermediate group granularity ( G ) and structural density allocation across GLUE benchmarks.
Strategy / Method
Config
Params
SST-2
CoLA
QNLI
RTE
Avg.
Severe Bottleneck (PAdapter)
G=10,d=5
101K
95.76
66.06
93.59
82.67
84.52
Aggressive Sharing (SAPE)
G=2,d=16
101K
96.10
68.37
93.79
87.36
86.41
Table 5: Comparison of parameter reduction strategies under an equivalent budget ( ∼101K parameters).
Masking Strategy
Dropout ( p )
SST-2
CoLA
QNLI
RTE
Avg.
Shared Mask (across group)
0.10
95.76
66.98
93.52
85.56
85.46
No Dropout
0.00
95.99
66.64
93.33
83.75
84.93
Independent Mask (SAPE)
0.10
96.10
68.37
93.79
87.36
86.41
Table 6: Ablation study on dropout masking mechanics within shared adapter layers.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch Size (Train / Eval)
32
Learning Rate
1×10−4
Optimizer
AdamW *
LR Scheduler
Linear *
Weight Decay
0.0 *
Warmup Ratio
0.0 *
Appendix
Table 10: Hyperparameter and architectural configurations for the SAPE topology ablation study on DeBERTa V3-base . All four routing strategies (Regular, Top-Slice, Bottom-Slice, and Sandwich) were trained under identical optimization conditions to ensure a fair parameter budget. Asterisks (*) denote implicit framework defaults.