A Spectral Framework for Universal Proxy Regularization in Model Merging
Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Computing and Information Technology, University of Wollongong, Australia · Nanyang Technological University, Singapore
Abstract
Model merging constructs multi-task models without expensive joint training but often suffers from parameter interference. Some solutions mitigate this by constraining finetuning using data from the other tasks to be merged, violating the core independence and privacy assumption of model merging. Recent work offers preliminary empirical evidence that a universal, task-agnostic proxy distribution can substitute for such data, but leaves open when and why it suffices. To this end, we propose a Spectral Framework for Universal Proxy Regularization, which characterizes the relationship between a proxy distribution and the distribution of other tasks to be merged through spectral coverage () and leakage (). Based on the framework, we derive a unified parameter interference bound: smaller provably yields lower parameter interference through better spectral alignment. To exploit such a proxy, we augment the training objective with a proxy regularizer. Evaluating this regularizer during fine-tuning requires a concrete mechanism. We study three instantiations, including (J) an exact evaluation via Jacobian-vector products, (F) a frozen curvature approximation using the Fisher information matrix, and (O) an output-drift formulation that constrains the model's functional deviation on the proxy directly. We prove that all three coincide under the NTK linearization, where existing analyses of such regularizers stop, and go further to characterize standard non-linear fine-tuning, bounding how far each instantiation departs from its linearized reference and obtain a three-way hierarchy, zero for (J), proxy-independent for (F), and proxy-quality-scaling for (O). Empirical validation on the commonly used vision-language model CLIP confirms these theoretical predictions.
Figures & tables
| Penalty | Mechanism | Additional per-step cost | Extra memory | Source of |
| (J) | exact quadratic form via JVPs | JVP + VJP | snapshot | |
| (F) | frozen | parameter-space matmuls | Fisher Matrix | curvature error |
| (O) | exact output drift | forward | snapshot | scaling as |
| Proxy | Mode | Absolute Accuracy (%) | ||
| (J) | (F) | (O) | ||
| ImageNet | Lin | 81.40 | 79.07 | 81.88 |
| Std | 83.73 | 77.03 | 82.24 | |
| (ref.) | ||||
| img-sub-u | Lin | 80.67 | 78.32 | 80.39 |
| Mode | Reg. | Step (ms) | Steady peak (MiB) | Prepare | |
| Time (s) | Peak (MiB) | ||||
| Standard | none | ||||
| (J) | |||||
| (F) | |||||
| (O) | |||||
| Linear | none | ||||
| Merging Method | Nonlin. FT | Nonlin. FT + (O) |
| Ties-Merging [ 14 ] | 72.73 | 81.20 |
| Task Singular Vectors [ 3 ] | 85.64 | 90.33 |
| Representation Surgery [ 35 ] | 80.70 | 85.35 |
| AdaMerging [ 2 ] (task-wise) | 70.71 | 82.74 |
| Proxy | Full | ||||
| Oracle Union | 84.43 | 84.87 | 85.06 | 85.18 | 85.14 |
| ImageNet | 80.79 | 81.23 | 81.37 | 81.64 | 81.40 |
| img-sub-u | 78.91 | 79.32 | 79.89 | 80.11 | 80.67 |
| sd-img | 80.35 | 80.95 | 81.04 | 81.49 | 81.46 |
| COCO | 79.87 | 80.54 | 80.71 | 81.05 | 81.65 |
| sd-wordnet | 80.72 | 81.12 | 81.25 | 81.40 | 81.19 |
| Proxy | |||||
| none | 74.86 | 75.26 | 75.09 | 75.12 | 68.23 |
| ImageNet | 81.40 | 82.35 | 82.19 | 82.17 | 76.37 |
| img-sub-u | 80.67 | 80.71 | 80.74 | 80.72 | 73.96 |
| sd-img | 81.46 | 81.28 | 81.29 | 81.23 | 75.94 |
| COCO | 81.65 | 81.31 | 81.28 | 81.28 | 74.95 |
| sd-wordnet | 81.19 | 81.65 | 81.63 | 81.52 | 76.63 |
| Method | Llama-3.2-1B | Llama-3.2-3B |
| AdamW | 41.07 | 49.39 |
| AdamW (J) | 42.69 | 51.69 |
| AdamW (O) | 43.02 | 51.20 |
| MergOPT | 42.75 | 50.26 |
| MergOPT (J) | 42.86 | 52.13 |
| MergOPT (O) | 44.72 | 51.46 |
| Method | Sec./step | Peak alloc. (GiB) |
| AdamW | 0.649 | 15.41 |
| AdamW | 0.862 (+32.8%) | 15.89 (+3.1%) |
| AdamW | 0.962 (+48.2%) | 32.75 (+112.5%) |
| Method | Inst. | Peer data | Specialist Avg. | Task Arithmetic | TIES-Merging | ||
| Abs. ( ) | Norm. ( ) | Abs. ( ) | Norm. ( ) | ||||
| Standard LoRA | – | No | 87.30 | 62.81 | 70.76 | 59.46 | 66.63 |
| OSRM [ 10 ] | – | Yes | 88.53 | 75.01 | 83.98 | 67.66 | 75.52 |
| Standard LoRA +(O) | (O) | No | 86.27 | 74.48 | 85.93 | 67.88 | 78.20 |
| Standard LoRA +(J) | (J) | No | 86.08 | 74.18 | 85.77 | 70.24 | 80.86 |
| Method | Sec./step | Peak alloc. (GiB) |
| Standard LoRA | 2.461 | 14.57 |
| Standard LoRA | 3.938 ( ) | 14.57 ( ) |
| Standard LoRA | 7.021 ( ) | 16.73 ( ) |
| Mean over ImageNet classes | Most similar img-sub-u class | ||||
| Target | All (1000) | Most similar (150) | img-sub-u (150) | img-sub-u class target label | |
| Cars | 0.618 | 0.713 | 0.496 | 0.560 | European polecat Volvo XC90 SUV 2007 |
| DTD | 0.720 | 0.828 | 0.520 | 0.645 | Gila monster scaly |
| EuroSAT | 0.698 | 0.815 | 0.491 | 0.624 | bustard pasture land |
| GTSRB | 0.692 | 0.817 | 0.473 | 0.592 | green mamba go straight (mandatory) |
| MNIST | 0.696 | 0.822 | 0.478 | 0.596 | military hat (bearskin or shako) 1 |
| Method | C-STANCE | FOMC | MeetingBank | ScienceQA | NumGLUE-cm | NumGLUE-ds | 20Minuten | Avg. ( ) |
| Pre-Trained | 0.3386 | 0.2581 | 0.2036 | 0.6780 | 0.1220 | 0.1646 | 0.3802 | 0.3064 |
| AdamW Fine-Tuned | 0.4980 | 0.5988 | 0.3707 | 0.8524 | 0.3902 | 0.5793 | 0.3880 | 0.5254 |
| MergOPT Fine-Tuned | 0.4957 | 0.6331 | 0.3158 | 0.8780 | 0.4390 | 0.5305 | 0.3829 | 0.5250 |
| AdamW | 0.4120 | 0.4899 | 0.2270 | 0.7245 | 0.2439 | 0.3963 | 0.3812 | 0.4107 |
| AdamW + (J) | 0.4210 | 0.5242 | 0.2213 | 0.7440 | 0.2927 | 0.4024 | 0.3827 | 0.4269 |
| AdamW + (O) | 0.4260 | 0.5464 | 0.2278 | 0.7405 | 0.2683 | 0.4207 | 0.3815 | 0.4302 |
| Method | C-STANCE | FOMC | MeetingBank | ScienceQA | NumGLUE-cm | NumGLUE-ds | 20Minuten | Avg. ( ) |
| Pre-Trained | 0.4082 | 0.3528 | 0.2054 | 0.8962 | 0.1707 | 0.2195 | 0.3857 | 0.3770 |
| AdamW | 0.4680 | 0.5565 | 0.2168 | 0.9110 | 0.4878 | 0.4268 | 0.3904 | 0.4939 |
| AdamW + (J) | 0.4870 | 0.5726 | 0.2303 | 0.9130 | 0.5610 | 0.4695 | 0.3846 | 0.5169 |
| AdamW + (O) | 0.4880 | 0.5565 | 0.2285 | 0.9160 | 0.5122 | 0.5000 | 0.3829 | 0.5120 |
| MergOPT | 0.4755 | 0.6028 | 0.2172 | 0.9155 | 0.4878 | 0.4268 | 0.3923 | 0.5026 |
| MergOPT + (J) | 0.4890 | 0.6169 | 0.2188 | 0.9155 | 0.5122 | 0.5061 | 0.3902 | 0.5213 |
| Stage | Method | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI | QNLI | RTE | Avg. |
| Standard LoRA [ 40 ] | 63.45 | 95.76 | 88.24 | 90.83 | 90.25 | 88.56 | 94.29 | 87.00 | 87.30 | |
| OSRM [ 10 ] | 66.93 | 96.56 | 90.44 | 90.72 | 90.20 | 90.27 | 95.73 | 87.36 | 88.53 | |
| Standard LoRA | 65.02 | 96.33 | 85.78 | 86.59 | 90.05 | 89.56 | 95.22 | 81.59 | 86.27 | |
| Specialist | Standard LoRA | 64.98 | 96.10 | 84.56 | 87.40 | 89.75 | 89.90 | 95.41 | 80.51 | 86.08 |
| Standard LoRA | 24.34 | 62.84 | 55.15 | 76.02 | 81.52 | 51.38 | 71.10 | 80.14 | 62.81 | |
| OSRM | 40.78 | 80.39 | 74.51 | 77.09 | 77.37 | 76.53 | 87.13 | 86.28 | 75.01 |
| Method | Role in this paper | Alone | + Ours |
| Task Arithmetic [ 5 ] | Merging rule, all experiments | 70.32 | 82.24 |
| TIES-Merging [ 14 ] | Merging rule, Tabs. V , X | 72.73 | 81.20 |
| DARE [ 15 ] | Related work | – | – |
| TSV [ 3 ] | Combined, Tab. V | 85.64 | 90.33 |
| KnOTS [ 16 ] | Related work | – | |
| AdaMerging [ 2 ] | Combined, Tab. V | 70.71 | 82.74 |
| Method | No peer-task info | Role in this paper | ViT-B/32 | Difference from ours |
| No explicit cross-task constraint: shape task vectors during fine-tuning, but do not penalize their effect on other tasks’ inputs. | ||||
| Nonlin. FT [ 5 ] | ✓ | Baseline, Tab. IV | 70.32 | Standard fine-tuning, the starting point of every row below. Ours adds the proxy penalty to this step. |
| TTA [ 6 ] | ✓ | Baseline, Tab. IV | 74.86 | Tangent-space fine-tuning improves disentanglement only implicitly; ours explicitly constrains where acts and is analyzed beyond linearization. TTA also serves as the linearized mode (Sec. VI-B ). |
| L-LoRA [ 17 ] | ✓ | Related work | – | Linearizes only the LoRA adapters (PEFT counterpart of TTA). |
| ATT-FT [ 7 ] | ✓ | Baseline, Tab. IV | 78.07 | Fine-tunes only the attention layers, which induces kernel-like behavior and improves disentanglement implicitly. Ours adds an explicit penalty and can be stacked on it (last row). |
| OrthoReg [ 4 ] | ✓ | Related work | Orthogonality prior on each layer’s update in weight space, without reference to inputs. | |
Model Merging
Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.
See all 214 papers