cs.CVMar 28, 2026
SaveDecompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression
Organizations: Boston University · NVIDIA, Boston University
Abstract
Parameter Recombination (PR) methods aim to efficiently compose the weights of a neural network for applications like Parameter-Efficient FineTuning (PEFT) and Model Compression (MC), among others. Most methods typically focus on one application of PR, which can make composing them challenging. For example, when deploying a large model you may wish to compress the model and also quickly adapt to new settings. However, PEFT methods often can still contain millions of parameters. This may be small compared to the original model size, but can be problematic in resource constrained deployments like edge devices, where they take a larger portion of the compressed model's parameters. To address this, we present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (CRISP), a general approach that seamlessly integrates multiple PR tasks within the same framework. CRISP accomplishes this by factorizing pretrained weights into basis matrices and their component mixing projections. Sharing basis matrices across layers and adjusting its size enables us to perform MC, whereas the mixer weight's small size (fewer than 200 in some experiments) enables CRISP to support PEFT. Experiments show CRISP outperforms methods from prior work capable of dual-task applications by 4-5% while also outperforming the state-of-the-art in PEFT by 1.5% and PEFT+MC combinations by 1%. Our code is available on the repository: https://github.com/appledora/CRISP-CVPR26.
Figures & tables
Figure 1: PR approach comparison. (a) Prior work in PR typically focuses on PEFT or MC alone [ 78 , 94 , 15 , 28 , 47 , 69 , 7 , 40 , 102 , 54 , 3 , 97 , 26 , 60 , 86 , 77 , 19 , 68 , 90 ] , which can result in efficient combinations when deployed together. (b) Our unified PR approach CRISP decomposes a pretrained models weights that support both MC and PEFT, enabling us to more effectively use parameter budgets even as tasks scale.
Figure 2: CRISP decomposes a pretrained weight matrix into a frozen shared basis and small, learnable mixer matrices, then retrofits these components back into the model (Sec. 3.2 ). Compression is achieved by reducing the basis size, while adaptation is enabled by fine-tuning only the lightweight, nonlinearly gated mixer matrices (Sec. 3.1 ) - allowing both Parameter Recombination (PR) applications to coexist within a single factorized structure with no redundant adapters).
| Method | Natural | Specialized | Structured | ||||||||||||||||||||
| CIFAR-100 | Cltch101 | DTD | Flowers102 | Pets | SVHN | Sun397 | Mean | Camelyon | EuroSAT | Resisc45 | Retinopathy | Mean | Clevr-Count | Clevr-Dist | DMLab | KITTI-Dist | dSpr-Loc | dSpr-Ori | sNORB-Azim | sNORB-Elev | Mean | Overall | |
| ViT-S/16 [ 13 ] | 42.1 | 84.4 | 62.6 | 97.3 | 87.2 | 31.1 | 43.8 | 64.0 | 74.8 | 87.8 | 75.2 | 70.8 | 77.1 | 34.9 | 29.8 | 34.8 | 45.9 | 14.5 | 15.1 | 11.1 | 20.0 | 25.7 | 50.7 |
| BoFT [ 48 ] | 34.4 | 86.1 | 60.5 | 97.3 | 82.2 | 61.4 | 35.8 | 65.3 | 73.5 | 89.6 | 73.3 | 68.4 | 76.1 | 31.2 | 45.1 | 36.0 | 47.5 | 22.6 | 16.3 | 12.6 | 25.6 | 29.6 | 52.6 |
| DoRA [ 47 ] | 38.5 | 86.9 | 60.6 | 97.2 | 85.9 | 52.7 | 33.2 | 65.0 | 72.9 | 86.0 | 75.3 | 66.9 | 75.2 | 46.1 | 45.7 | 34.1 | 41.6 | 21.4 | 21.9 | 11.2 | 25.1 | 30.8 | 52.8 |
| IA3 [ 45 ] | 11.3 | 24.5 | 25.9 | 41.1 | 12.8 | 34.7 | 7.6 | 22.5 | 71.2 | 75.0 | 43.9 | 71.9 | 65.4 | 35.6 | 50.9 | 24.4 | 39.5 | 22.6 | 10.4 | 7.8 | 18.2 | 26.1 | 33.1 |
| LoRA [ 28 ] | 52.8 | 85.0 | 64.3 | 98.1 | 88.4 | 55.9 | 45.4 | 69.9 | 75.6 | 90.4 | 79.0 | 69.3 | 78.5 | 45.0 | 47.6 | 39.5 | 52.7 | 19.8 | 23.1 | 11.7 | 18.2 | 32.1 | 55.8 |
Table 1: PEFT performance on VTAB-1K [ 93 ] across 19 tasks grouped into Natural (7), Specialized (4), and Structured (8). CRISP achieves state-of-the-art overall accuracy while tuning 28% fewer parameters than all baselines ( vs. of the base model). It is particularly strong on Structured tasks, achieving the best result on the majority of benchmarks. Bold indicates best performance per task.
Figure 3: PEFT performance using a ViT-S/16 across a range of trainable parameter budgets averaged over three datasets: FGVC-Aircraft [ 50 ] , CIFAR-100 [ 34 ] and CUB-200-2011 [ 74 ] . CRISP consistently outperforms prior work in all settings.
| Compression | PEFT | Flowers | Aircraft | Scene | CFR100 | CFR10 | Birds | Avg |
| ViT-B/16 [ 13 ] | – | 96.7 | 70.9 | 84.5 | 76.3 | 97.0 | 84.6 | 85.0 |
| Compressed Performance | ||||||||
| SVD | – | 81.8 | 43.1 | 65.6 | 55.4 | 83.0 | 39.3 | 61.3 |
| Isomorph. Prune. [ 16 ] | – | 94.7 | 61.5 | 77.6 | 76.3 | 93.2 | 70.0 | 78.8 |
| RDHP [ 84 ] | – | 95.3 | 63.8 | 74.7 | 78.1 | 89.2 | 61.7 | 77.1 |
| DGMR [ 67 ] | – | 96.4 | 71.6 | 79.8 | 76.9 | 92.1 | 74.3 | 81.9 |
Table 2: ViT-B/16 [ 13 ] compression at 50% parameter reduction across six fine-grained benchmarks. Upper: post-compression accuracy with classifier-only adaptation; CRISP leads all baselines using only 2% of ImageNet-1K [ 11 ] for distillation. Lower: compressed backbones used as initialization for PEFT; CRISP achieves state-of-the-art, outperforming the best pruning+PEFT combination by about 1 point and RECAST with coefficient tuning by 5 points, suggesting that compression quality directly bounds downstream task adaptability.
| BQ | PIQ | Hell. | Wino | ARC-e | ARC-c | OBQ | Avg. | |
| LLaMA3.2-1B [ 22 ] | 63.9 | 74.4 | 47.7 | 60.0 | 65.4 | 31.3 | 26.4 | 52.7 |
| SVD | 38.1 | 53.2 | 26.0 | 47.9 | 26.1 | 21.5 | 15.6 | 32.6 |
| Basis-Sharing [ 77 ] | 38.2 | 54.9 | 26.5 | 50.5 | 26.9 | 20.2 | 14.6 | 33.1 |
| PruneNet [ 65 ] | 51.6 | 54.7 | 26.3 | 48.6 | 28.6 | 17.8 | 16.8 | 34.9 |
| DFJR [ 57 ] | 37.8 | 53.7 | 26.1 | 49.6 | 26.1 | 19.3 | 14.2 | 32.4 |
| CRISP (ours) | 60.2 | 57.5 | 27.7 | 50.6 | 34.6 | 20.1 | 15.2 | 38.0 |
Table 3: LLaMA3.2-1B [ 22 ] compression at 30% parameter reduction across seven commonsense reasoning benchmarks. CRISP’s 3% average gain shows it can perform well on LLMs, demonstrating an ability generalize across architectures we found many prior works lack.
Figure 4: Comparing ImageNet [ 11 ] performance with and without 8-bit PTQ [ 83 ] compression. We find CRISP accurately reproduces the original model’s performance while also demonstrating effective compositionality with other compression techniques.
| Type | Birds | CIF100 | ACraft | Av. |
| No regularization | 81.5 | 86.0 | 77.9 | 81.8 |
| L2-Norm | 82.6 | 86.5 | 77.0 | 82.0 |
| Orthogonal | 84.2 | 86.9 | 76.1 | 82.3 |
| Spectral Norm | 83.5 | 85.7 | 77.4 | 82.1 |
| ReLU | 58.7 | 64.0 | 61.9 | 61.5 |
| GELU | 80.1 | 85.9 | 79.0 | 81.6 |
Table 4: Ablation on the regularization strategy for the mixer matrix using ViT-S/16 [ 13 ] averaged over CUB-Birds, CIFAR-100, and FGVC-Aircraft. (a) Explicit regularization provides modest gains but requires additional hyperparameters. (b) Activation-based constraints offer a parameter-free alternative, where CRISP’s use of SiLU’s smooth gating matches explicit regularization without introducing new hyperparameters.
Figure 5: Impact of mixer matrix dimensions on model capacity and performance. (a) Fixed columns ( ): Increasing rows reduces parameters but collapses accuracy. (b) Fixed rows ( ): Increasing columns scales capacity and recovers performance. Results across CIFAR-100, CUB-Birds, and FGVC-Aircraft demonstrate that basis capacity (columns) is the dominant factor for maintaining model quality, while coefficient expressivity (rows) plays a secondary role. Red dotted line: original model.
| Method | Tr.Thr. | In.Thr. | Lat. | Mem. |
| samp/s | samp/s | ms | GB | |
| LoRA [ 28 ] | 165.2 | 575 | 1.74 | 4.58 |
| DoRA [ 47 ] | 178.3 | 726 | 1.38 | 3.98 |
| BoFT [ 48 ] | 74.7 | 155 | 6.43 | 5.24 |
| IA3 [ 45 ] | 156.8 | 629 | 1.59 | 6.87 |
| RoAD [ 41 ] | 131.0 | 479 | 2.09 | 8.40 |
Table 5: Efficiency using a NVIDIA L40S for task adaptation on ViT-B/16 [ 13 ] backbone under matched finetuning parameter budgets ( 150K–240K trainable parameters, excluding head).
| Dataset | #Cat | #Train | #Val | #Test |
| CIFAR100 | 100 | 800/1000 | 200 | 10000 |
| Caltech101 | 102 | 6084 | – | – |
| DTD | 47 | 1880 | – | – |
| Flower102 | 102 | 6149 | – | – |
| Pets | 37 | 3669 | – | – |
| SVHN | 10 | 26032 | – | – |
Table 6: Details of VTAB-1K [ 93 ] Benchmark used for PEFT on ViT-S/16 [ 13 ] .
| Dataset | Classes | #Sample |
| Oxford Flowers [ 55 ] | 102 | 6553 |
| FGVC Aircrafts [ 50 ] | 55 | 10001 |
| MIT Scenes [ 59 ] | 67 | 15614 |
| CIFAR100 [ 34 ] | 100 | 60000 |
| CIFAR10 [ 34 ] | 10 | 60000 |
| CUBs (Birds) [ 74 ] | 200 | 11789 |
Table 7: Details on the Benchmarking Datasets used for fine-tuning Compressed ViT-B/16 [ 13 ]
Figure 6: Impact of regularization constraint placement across PRE, POST, and TEMP configurations. PRE (our method) achieves the most consistent performance, while ReLU causes severe degradation due to weight sparsification.
Figure 7: Effect of reconstruction loss functions during neural mimicry. We compare four loss functions (Huber, Smooth-L1, MSE, L1) used in the neural mimicry stage (Equation 5 of main paper) for retrofitting pretrained weights into CRISP’s basis-mixer decomposition.
Figure 8: Robustness to initialization methods. We evaluate four standard initialization schemes (Uniform, Kaiming, Xavier, Orthogonal) for the mixer matrices during both neural mimicry retrofitting and subsequent task adaptation.
| Compression | Params(%) | Flowers | Aircraft | Scene | CFR100 | CFR10 | Birds | Avg |
| Neural Mimicry (Eq. 5 of main paper) | 44M | 83.9 | 56.3 | 51.2 | 49.8 | 79.4 | 26.3 | 57.8 |
| Distillation Loss | 44M | 99.0 | 89.5 | 81.8 | 86.2 | 97.4 | 79.1 | 88.8 |
| w/o SVD Init | 44M | 98.8 | 85.9 | 77.4 | 82.0 | 95.3 | 71.6 | 85.1 |
Table 8: Compression results on ViT-B/16 [ 13 ] with 50% weight compression across six diverse tasks (See Table 7 ). We find that using the distillation loss with SVD initialization described in Sec. 3.2 of our paper provides best performance.
| Compression | PEFT | Params(%) | Flowers | Aircraft | Scene | CFR100 | CFR10 | Birds | Avg |
| ViT-B/16 [ 13 ] | – | 86M | 96.7 | 70.9 | 84.5 | 76.3 | 97.0 | 84.6 | 85.0 |
| SVD | – | 21M | 70.7 | 25.7 | 33.2 | 39.6 | 62.5 | 8.6 | 40.0 |
| RECAST [ 71 ] | – | 21M | 90.0 | 59.1 | 67.2 | 67.1 | 85.3 | 52.9 | 70.2 |
| CRISP (ours) | – | 21M | 95.1 | 73.0 | 77.7 | 74.1 | 89.7 | 73.6 | 80.5 |
| SVD | Eigenvalues | 21M | 87.8 | 55.9 | 62.1 | 65.0 | 85.3 | 40.1 | 66.0 |
| RECAST [ 71 ] | RECAST | 21M | 94.7 | 67.4 | 71.0 | 71.4 | 89.5 | 58.9 | 75.5 |
Table 9: Compression results on ViT-B/16 [ 13 ] at 75% parameter reduction evaluated on six fine-grained classification benchmarks (see Tab. 2 of the main paper for 50% reduction). Upper section: post-compression accuracy with only classifier adaptation. Lower section: MC+PEFT combinations demonstrate compressed models as initialization for downstream tasks. We find that CRISP outperforms prior work by up to 11%.
| Compression | PEFT | Accuracy ( ) | |||||||
| BQ | PIQ | Hell. | Wino | ARC-e | ARC-c | OBQ | Avg. | ||
| LLaMA3.2-1B [ 22 ] | – | 60.0 | 90.0 | 30.0 | 70.0 | 70.0 | 30.0 | 10.0 | 51.4 |
| Basis-Sharing [ 77 ] | LoRA [ 28 ] | 37.8 | 52.8 | 26.8 | 48.6 | 28.2 | 19.3 | 16.2 | 32.8 |
| Basis-Sharing [ 77 ] | DoRA [ 47 ] | 37.8 | 53.8 | 26.8 | 51.0 | 28.9 | 19.2 | 13.2 | 33.0 |
| DFJR [ 57 ] | LoRA [ 28 ] | 48.9 | 53.5 | 26.4 | 50.3 | 28.6 | 20.6 | 14.4 | 34.7 |
| DFJR [ 57 ] | DoRA [ 47 ] | 57.2 | 54.7 | 27.3 | 50.9 | 29.2 | 19.8 | 13.6 | 36.1 |
Table 11: LLaMA3.2-1B [ 22 ] MC+PEFT results at 30% parameter reduction across seven commonsense reasoning benchmarks. All baselines pair a compression method with a separate PEFT method, whereas CRISP compresses and adapts within the same factorized framework using lightweight coefficient tuning.
More in this topic
Neural Network Compression
Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.
See all 216 papers