cs.LGOct 14, 2026

Your Transformer Is Secretly Elastic

Authors: Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, Cees G. M. Snoek

Organizations: University of Amsterdam · TNO, Intelligent Imaging

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to 1.80×1.80\times higher throughput and 1.26×1.26\times lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.

Figures & tables

Figure 1: Differentiable soft pruning. Initial importance scores for MLP neurons and attention heads are transformed into learnable log-scores and median-centered by type, aligning their reference points without changing within-type rankings. At each optimization step, the joint ranking determines a threshold κb\kappa_{b} and soft mask MbM_{b} for each sparsity target b∈Bb\in\mathcal{B} . We run the same calibration batch through the frozen model under each mask, average losses across targets, and update only the log-scores. Mask gradients concentrate near each threshold, so multiple targets expose different regions of the ranking to useful gradients. The optimized ranking can then be hard-pruned to any inference budget without further optimization.
Figure 2: Our method yields smooth degradation curves across vision and text tasks. Performance versus compute for our method and six baselines from dense to 60% sparsity. Curve labels indicate FLOP sparsity. Top: ImageNet-1k top-1 k-NN accuracy. Bottom left: Pascal VOC 2012 Hummingbird mIoU. Bottom right: mean MTEB score. Baselines remain competitive under shallow pruning, but our method retains higher performance as sparsity increases, with the largest gains under moderate and deep pruning.
Figure 3: Larger vision models tolerate more pruning. Performance versus compute for our method and baselines from dense to 60% sparsity. Curve labels indicate FLOP sparsity. Top: ImageNet-1k top-1 k-NN accuracy for DeiT-III and DINOv3. Bottom: mean MTEB score for BGE and Qwen3. Size helps BGE modestly and Qwen3 inconsistently. Our advantage grows with size except on DeiT-III, where SnapViT remains competitive.
Figure 4: Our method better preserves compatibility with a fixed dense index. Performance versus compute for our method and baselines from dense to 60% sparsity, with the index encoded once by the dense model and queries encoded by its pruned counterparts. Curve labels indicate FLOP sparsity. Top: ImageNet-1k top-1 k-NN accuracy. Bottom: mean BEIR nDCG@10. Baselines remain competitive under shallow pruning, but our method supports deeper query encoder pruning without rebuilding the index.
Optimization GranularityParametersImageNet-1k
None029.4
Sublayer2429.4
MLP Sublayer + Head (SnapViT) †\dagger15634.6
8 Groups + Head24036.3
64 Groups + Head91243.3
512 Groups + Head6,28851.9
Table 1: Per-unit optimization mitigates the diagonal approximation. ImageNet-1k top-1 k-NN accuracy for AugReg ViT-B/16 at 50% sparsity under different score parametrizations. Groups are neuron groups per MLP block. All optimized variants use equal compute. Per-unit scores achieve the highest accuracy.
 Sparsity
 20% / 28.2 G40% / 21.2 G60% / 14.2 G
ScorePlainOursPlainOursPlainOurs
Random58.276.8 +18.614.763.2 +48.57.935.5 +27.6
Magnitude7.377.0 +69.74.263.7 +59.52.533.8 +31.3
Wanda7.277.8 +70.66.263.1 +56.94.832.6 +27.8
SNIP Mag.78.279.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.5}}37.869.4 +31.612.443.3 +30.9
Table 2: Robustness to score initialization. ImageNet-1k top-1 k-NN accuracy for AugReg ViT-B/16 at 20%, 40%, and 60% sparsity, before (Plain) and after (Ours) ranking optimization. Optimization improves all score initializations, gaining up to 70.6% for Wanda at 20% sparsity.
SparsitySingleElastic
20% / 28.2 G79.780.1 +0.4
40% / 21.2 G67.269.1 +1.9
60% / 14.2 G41.645.2 +3.6
Table 3: Joint optimization across sparsity targets improves performance. ImageNet-1k top-1 k-NN accuracy for AugReg ViT-B/16. Joint optimization over 14 targets (Elastic) outperforms single-target optimization (Single) at all three sparsities.
Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetClassesAvg. Native ResolutionMin Native ResolutionMax Native ResolutionTraining SamplesTest Samples
DTD Textures ( Cimpoi et al., 2014 )47500×453500\times 453300×231300\times 231800×701800\times 70118801880
FGVC Aircraft ( Maji et al., 2013 )1001104×7511104\times 751775×430775\times 4301600×11881600\times 118833343333
EuroSAT ( Helber et al., 2018 )1064×6464\times 6464×6464\times 6464×6464\times 64216005400
CIFAR 10 ( Krizhevsky, 2009 )1032×3232\times 3232×3232\times 3232×3232\times 325000010000
CIFAR 100 ( Krizhevsky, 2009 )10032×3232\times 3232×3232\times 3232×3232\times 325000010000
Oxford-IIIT Pets ( Parkhi et al., 2012 )37431×383431\times 383114×108114\times 1083264×26063264\times 260636803669
Appendix
Table 4: Image classification datasets. Summary table of the image classification datasets.
Figure 5: Ranking optimization changes both cross-block sparsity allocation and within-block selection. Retained MLP neurons and attention heads at 50% sparsity for AugReg, DINOv3, and SigLIPv2 ViT-B/16. The top rows compare per-block allocations from plain and optimized scores with uniform pruning. The bottom rows use the optimized per-block allocation and divide its retained units into those also selected by the plain scores and those newly selected after ranking optimization. Newly retained units appear even where block-level allocations remain nearly unchanged, showing that ranking optimization also reorders units within blocks rather than merely redistributing capacity across them.
Figure 6: Our method retains higher accuracy under deep pruning on DeiT-B/16. Top-1 k-NN (top) and linear-probing (bottom) accuracy versus GFLOPs on ImageNet-1k and the transfer average. Most baselines are competitive under shallow pruning, but degrade more sharply than our method as sparsity increases.
Figure 7: Our method degrades more smoothly under linear probing on AugReg ViT-B/16. Top-1 accuracy in linear classification on ImageNet-1k and transfer datasets for AugReg ViT-B/16 models pruned with our method and the baselines. Compared with the corresponding k-NN results in Figure 9 , linear probes retain more accuracy under deep pruning.
Figure 8: Our method outperforms baselines in linear classification under deep pruning on large DINOv3 and SigLIPv2 models. Top-1 accuracy in linear classification on ImageNet-1k and transfer datasets for DINOv3 ViT-H+/16 (top) and SigLIPv2 ViT-G/16 (bottom), comparing our method with compatible baselines. Notably, the linear accuracy on ImageNet-1k for DINOv3 is essentially flat at up to 50% sparsity.
Figure 9: Larger AugReg models tolerate more pruning. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for AugReg models pruned with our method and the baselines.
Figure 10: Larger DeiT-III models tolerate more pruning. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for DeiT-III models pruned with our method and the baselines. At 60% sparsity, ViT-H/14 loses 10.4% on ImageNet-1k and 2.73% on the transfer average, while removing approximately 378M parameters.
Figure 11: Larger DINOv3 models tolerate more pruning. Top-1 k-NN accuracy on ImageNet-1k and the transfer average for DINOv3 models pruned with our method and the baselines. Our method yields smoother degradation on both models. Pruning the ViT-H+/16 to 60% sparsity removes 512M parameters, while reducing k-NN accuracy by 5.6% on ImageNet-1k and 6.4% on the transfer average.
Figure 12: Larger SigLIPv2 models tolerate more pruning. Top-1 k-NN accuracy on ImageNet-1k and the transfer average for SigLIPv2 ViT-B/16 and ViT-G/16 models pruned with our method and the baselines. Our method yields smoother degradation on both models. Pruning the ViT-G/16 to 50% sparsity removes 573M parameters, while reducing k-NN accuracy by 10.5 percentage points on ImageNet-1k and 12.9 percentage points on the transfer average.
Figure 13: Pruned larger models can outperform smaller dense models. ImageNet-1k top-1 k-NN accuracy versus GFLOPs for the six-model DINO WebSSL family, pruned with our method from dense to 60% sparsity. Pruned 3B and 7B models pareto-dominate the dense 2B and 5B anchors.
Figure 14: AugReg ViT-B/16 degrades unevenly across tasks, with fine-grained tasks being the most sensitive. Top-1 k-NN accuracy versus GFLOPs curves on ImageNet-1k and the nine transfer datasets for AugReg ViT-B/16 pruned from 10% to 60% sparsity with our method and the baselines. At 50% sparsity, our method loses 24.3% on ImageNet-1k and over 30% on CIFAR-100, Flowers 102, and Food 101, but at most 8.1% on Pets, EuroSAT, and RESISC45. Our method leads the baselines under moderate and deep pruning, except on EuroSAT, where SnapViT and OPTIN retain higher accuracy.
Figure 15: DINOv3 ViT-H+/16 is nearly lossless at 50% sparsity, and improves on Flowers 102. Top-1 k-NN accuracy versus GFLOPs curves on ImageNet-1k and the nine transfer datasets for DINOv3 ViT-H+/16 pruned from 10% to 60% sparsity with our method and the baselines. At 50% sparsity, our method loses 1.8% on ImageNet-1k, and between 0.5% on Pets and 6.2% on CIFAR-100 on the transfer datasets, while Flowers 102 improves from 90.2% to 97.9%. At the same sparsity, it leads SnapViT, the runner-up, by 42.9% on ImageNet-1k, and from 40% onward it matches or exceeds every baseline on every dataset.
Figure 16: SigLIPv2 ViT-G/16 degrades unevenly at 50% sparsity, with fine-grained tasks being the most sensitive. Top-1 k-NN accuracy versus GFLOPs curves on ImageNet-1k and the nine transfer datasets for SigLIPv2 ViT-G/16 pruned from 10% to 60% sparsity with our method and the baselines. At 50% sparsity, our method loses 10.5% on ImageNet-1k, and between 0.8% on EuroSAT and 39.6% on Aircraft on the transfer datasets. Under shallow pruning, VBP and SnapViT lead on Aircraft, by 12.9% and 5.4% at 20% sparsity, but from 30% onward our method leads every baseline on every dataset.
Figure 17: Our method preserves patch-level representations. In-context semantic segmentation mIoU on Pascal VOC 2012 for AugReg ViT-B/16 models pruned with our method and the baselines. Our method retains performance under pruning, improving on the runner-up, OPTIN, by up to 25.1% at 50% sparsity.
Figure 18: Our method preserves DINOv3’s patch-level representations under pruning. In-context semantic segmentation mIoU on Pascal VOC 2012 for DINOv3 ViT-B/16 models pruned with our method and the baselines. Our method degrades smoothly and consistently outperforms each baseline at comparable compute.
   Sparsity
BackboneMethodDense10%20%30%40%50%60%
BGE-SSnapViT64.363.4 −0.90{}^{-0.9\phantom{0}}62.4 −1.90{}^{-1.9\phantom{0}}60.5 −3.80{}^{-3.8\phantom{0}}57.6 −6.70{}^{-6.7\phantom{0}}54.6 −9.70{}^{-9.7\phantom{0}}51.8 -12.5
 SlimGPT 63.2 −1.10{}^{-1.1\phantom{0}}61.2 −3.10{}^{-3.1\phantom{0}}58.4 −5.90{}^{-5.9\phantom{0}}54.7 −9.60{}^{-9.6\phantom{0}}48.9 -15.441.8 -22.5
 Ours 63.7 −0.60{}^{-0.6\phantom{0}}62.3 −2.00{}^{-2.0\phantom{0}}60.3 −4.00{}^{-4.0\phantom{0}}58.4 −5.90{}^{-5.9\phantom{0}}56.7 −7.60{}^{-7.6\phantom{0}}54.3 -10.0
BGE-BSnapViT65.265.0 −0.20{}^{-0.2\phantom{0}}64.2 −1.00{}^{-1.0\phantom{0}}63.2 −2.00{}^{-2.0\phantom{0}}60.1 −5.10{}^{-5.1\phantom{0}}55.6 −9.60{}^{-9.6\phantom{0}}49.9 -15.3
 SlimGPT 63.8 −1.40{}^{-1.4\phantom{0}}61.2 −4.00{}^{-4.0\phantom{0}}57.1 −8.10{}^{-8.1\phantom{0}}53.4 -11.840.9 -24.340.6 -24.6
Appendix
Table 5: Elastic pruning generalizes to text. MTEB score for dense and elastically pruned models across the BGE, E5, E5-Mistral, and Qwen3 families. Shallow pruning is nearly free: at 20% sparsity, no model loses more than 3.3 points, and the E5 models lose under 1.0 points. Under deep pruning, bidirectional encoders degrade more gracefully than decoders: at 60% sparsity, BGE and E5 lose between 5.1 and 10.0 points, while E5-Mistral and Qwen3 lose between 11.5 and 19.5.
Dataset ↓\downarrowMethod ↓\downarrowDense ↓\downarrowAttentionBlocksMLP
EffiR Variant →\rightarrowA8A16B8B16M8M16
Params, EffiR/Ours (B) →\rightarrow6.8/6.46.4/6.45.4/5.73.6/3.65.7/5.74.3/4.3
ArguAnaEffiR †61.655.30.139.09.359.08.4
 Ours61.759.959.960.153.060.157.1
ClimateEffiR †37.532.82.21.23.939.09.8
FEVEROurs38.438.438.436.121.436.128.2
Appendix
Table 6: Our elastic pruning preserves retrieval performance better than coarse layer dropping. nDCG@10 on 13 BEIR datasets, with queries and documents encoded by the same pruned model. We compare our elastically pruned E5-Mistral 7B models against similarly sized layer-dropping baselines from EffiR, both without retraining. A kk , B kk , and M kk denote dropping kk attention sublayers, transformer blocks, and MLP sublayers, respectively. Our mean score is higher in every comparison. At 4.3B parameters, our method achieves 49.9 versus 24.7 for dropping 16 MLP sublayers, a gain of 25.2 points.
Dataset ↓\downarrowMethod ↓\downarrowDense ↓\downarrowAttentionMLPMixedFull
EffiR Variant →\rightarrowA8A16M8M16M20M24M16+A8EffiR
Params, EffiR/Ours (B) →\rightarrow6.8/6.46.4/6.45.7/5.74.3/4.33.6/3.62.9/3.04.0/4.33.4/3.6
ArguAnaEffiR †58.256.147.456.954.551.445.653.252.8
 Ours61.759.959.960.157.153.050.757.153.0
ClimateEffiR †30.829.929.430.431.631.928.927.930.4
FEVEROurs38.438.438.436.128.221.421.128.221.4
Appendix
Table 7: Retraining-free pruning is competitive with retrieval-fine-tuned models at moderate sparsity. nDCG@10 on 13 BEIR datasets, with queries and documents encoded by the same pruned model. We compare our retraining-free pruning of E5-Mistral 7B against similarly sized EffiR models, obtained by pruning Mistral 7B and fine-tuning on MS MARCO. A kk and M kk denote dropping kk attention and MLP sublayers, respectively, and EffiR denotes the full coarse-to-fine method. Our mean score is higher down to 5.7B parameters. At 4.3B, our method achieves 49.9 versus 55.3 for the fine-tuned baseline, a gap of only 5.4 points without any weight updates. On ArguAna, our method leads across all compared sizes.
   Sparsity
BackboneMethodDense10%20%30%40%50%60%
BGE-SSnapViT67.667.2 −0.40{}^{-0.4\phantom{0}}66.5 −1.10{}^{-1.1\phantom{0}}65.5 −2.10{}^{-2.1\phantom{0}}64.0 −3.60{}^{-3.6\phantom{0}}62.5 −5.10{}^{-5.1\phantom{0}}60.4 −7.20{}^{-7.2\phantom{0}}
 SlimGPT 67.2 −0.40{}^{-0.4\phantom{0}}65.9 −1.70{}^{-1.7\phantom{0}}63.8 −3.80{}^{-3.8\phantom{0}}61.3 −6.30{}^{-6.3\phantom{0}}57.0 -10.650.9 -16.7
 Ours 67.3 −0.30{}^{-0.3\phantom{0}}66.7 −0.90{}^{-0.9\phantom{0}}65.6 −2.00{}^{-2.0\phantom{0}}64.5 −3.10{}^{-3.1\phantom{0}}63.3 −4.30{}^{-4.3\phantom{0}}61.9 −5.70{}^{-5.7\phantom{0}}
BGE-BSnapViT68.668.4 −0.20{}^{-0.2\phantom{0}}68.0 −0.60{}^{-0.6\phantom{0}}67.4 −1.20{}^{-1.2\phantom{0}}65.9 −2.70{}^{-2.7\phantom{0}}63.0 −5.60{}^{-5.6\phantom{0}}59.4 −9.20{}^{-9.2\phantom{0}}
 SlimGPT 67.6 −1.00{}^{-1.0\phantom{0}}65.8 −2.80{}^{-2.8\phantom{0}}63.6 −5.00{}^{-5.0\phantom{0}}62.3 −6.30{}^{-6.3\phantom{0}}53.2 -15.452.9 -15.7
Appendix
Table 8: Non-retrieval tasks are robust to pruning. MTEB score excluding retrieval tasks for dense and elastically pruned BGE, E5, E5-Mistral, and Qwen3 models. At 60% sparsity, BGE-L and E5-L lose 5.3 and 4.3 points, against 19.6 and 8.0 on retrieval tasks as shown in Table 9 .
   Sparsity
BackboneMethodDense10%20%30%40%50%60%
BGE-SSnapViT53.951.8 −2.10{}^{-2.1\phantom{0}}49.6 −4.30{}^{-4.3\phantom{0}}45.2 −8.70{}^{-8.7\phantom{0}}37.7 -16.230.3 -23.625.3 -28.6
 SlimGPT 50.9 −3.00{}^{-3.0\phantom{0}}46.7 −7.20{}^{-7.2\phantom{0}}41.6 -12.334.2 -19.723.8 -30.113.4 -40.5
 Ours 52.6 −1.30{}^{-1.3\phantom{0}}48.5 −5.40{}^{-5.4\phantom{0}}43.9 -10.039.6 -14.336.4 -17.530.6 -23.3
BGE-BSnapViT54.854.2 −0.60{}^{-0.6\phantom{0}}52.3 −2.50{}^{-2.5\phantom{0}}50.2 −4.60{}^{-4.6\phantom{0}}42.4 -12.432.8 -22.020.5 -34.3
 SlimGPT 52.1 −2.70{}^{-2.7\phantom{0}}47.2 −7.60{}^{-7.6\phantom{0}}36.9 -17.925.9 -28.92.6 -52.22.2 -52.6
Appendix
Table 9: Retrieval is sensitive to deep pruning. MTEB retrieval score for dense and elastically pruned BGE, E5, E5-Mistral, and Qwen3 models. Retrieval degrades faster than other tasks: at 60% sparsity, BGE-L loses 19.6 points and Qwen3 8B 22.1. On the other hand, models tolerate shallow pruning, and E5-B slightly exceeds its dense score at 20% sparsity.
Figure 19: Elastic pruning extends the performance-compute frontier of dense text encoders. MTEB score versus GFLOPs for BGE, E5, and Qwen3 encoders, from dense to 60% sparsity. Dashed lines indicate the pareto frontier. GFLOPs are shown on a logarithmic scale.
   Sparsity
BackboneMethodDense10%20%30%40%50%60%
BGE-SSnapViT51.750.0 −1.70{}^{-1.7\phantom{0}}46.5 −5.20{}^{-5.2\phantom{0}}41.6 -10.126.6 -25.114.2 -37.54.9 -46.8
 SlimGPT 48.8 −2.90{}^{-2.9\phantom{0}}41.6 -10.128.5 -23.29.8 -41.92.4 -49.30.2 -51.5
 Ours 51.4 −0.30{}^{-0.3\phantom{0}}48.3 −3.40{}^{-3.4\phantom{0}}44.5 −7.20{}^{-7.2\phantom{0}}40.0 -11.736.7 -15.032.0 -19.7
BGE-BSnapViT53.252.9 −0.30{}^{-0.3\phantom{0}}51.4 −1.80{}^{-1.8\phantom{0}}49.6 −3.60{}^{-3.6\phantom{0}}45.3 −7.90{}^{-7.9\phantom{0}}29.7 -23.512.6 -40.6
 SlimGPT 51.4 −1.80{}^{-1.8\phantom{0}}48.3 −4.90{}^{-4.9\phantom{0}}43.0 -10.232.9 -20.32.6 -50.60.6 -52.6
Appendix
Table 10: Our method preserves elastic retrieval performance across all nine text encoders. Mean nDCG@10 on the 15 BEIR tasks across all nine text encoders, where documents are encoded by the dense model and queries by its pruned counterparts. Our method outperforms both baselines on every backbone at 50% and 60% sparsity.
 Sparsity
 10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
MethodPlainCorrectedPlainCorrectedPlainCorrectedPlainCorrectedPlainCorrectedPlainCorrected
SparseGPT78.882.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.6}}64.279.3 +15.136.572.2 +35.720.359.9 +39.614.142.4 +28.310.229.1 +18.9
FPTP82.082.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.1}}78.579.5 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.0}}71.376.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.9}}44.357.0 +12.717.219.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.1}}12.613.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.7}}
SnapViT82.282.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.6}}77.581.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.5}}70.178.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.3}}57.773.0 +15.328.463.3 +34.917.747.5 +29.8
Ours82.682.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.2}}80.181.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.9}}75.778.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.4}}69.172.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.5}}59.364.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.0}}45.254.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.8}}
Appendix
Table 11: Our method is competitive against corrected baselines, and is compatible with weight correction. Top-1 k-NN accuracy on ImageNet-1k for pruned AugReg ViT-B/16, without (Plain) and with (Corrected) post-pruning weight correction. Uncorrected, our method outperforms SparseGPT and FPTP, only trailing corrected SnapViT by at most 4.0%. Corrected, it matches SnapViT at up to 40% sparsity and surpasses it beyond.
  Sparsity
MethodCorrection10% / 398.6 G20% / 354.4 G30% / 310.0 G40% / 265.8 G50% / 221.7 G60% / 177.5 G
SparseGPT✔85.285.180.669.655.438.3
SnapViT✔85.285.285.083.576.861.2
Ours✘85.185.185.184.783.379.4
Appendix
Table 12: Our correction-free method matches or outperforms weight-correction methods on DINOv3 ViT-H+/16. Top-1 k-NN accuracy on ImageNet-1k for DINOv3 ViT-H+/16 pruned with corrected SparseGPT and SnapViT, versus our correction-free method. At 10% and 20% sparsity, our method remains within 0.1% of the best corrected baseline. Our advantage grows with sparsity, to up to 18.2% over corrected SnapViT, the runner-up, at 60%.
   Sparsity
BackboneMethodDense10%20%30%40%50%60%
BGE-SSlimGPT64.363.2 −1.10{}^{-1.1\phantom{0}}61.2 −3.10{}^{-3.1\phantom{0}}58.4 −5.90{}^{-5.9\phantom{0}}54.7 −9.60{}^{-9.6\phantom{0}}48.9 -15.441.8 -22.5
 SlimGPT (Corrected) 63.6 −0.70{}^{-0.7\phantom{0}}62.5 −1.80{}^{-1.8\phantom{0}}60.4 −3.90{}^{-3.9\phantom{0}}58.3 −6.00{}^{-6.0\phantom{0}}54.0 -10.348.3 -16.0
 Ours 63.7 −0.60{}^{-0.6\phantom{0}}62.3 −2.00{}^{-2.0\phantom{0}}60.3 −4.00{}^{-4.0\phantom{0}}58.4 −5.90{}^{-5.9\phantom{0}}56.7 −7.60{}^{-7.6\phantom{0}}54.3 -10.0
BGE-BSlimGPT65.263.8 −1.40{}^{-1.4\phantom{0}}61.2 −4.00{}^{-4.0\phantom{0}}57.1 −8.10{}^{-8.1\phantom{0}}53.4 -11.840.9 -24.340.6 -24.6
 SlimGPT (Corrected) 64.1 −1.10{}^{-1.1\phantom{0}}63.1 −2.10{}^{-2.1\phantom{0}}61.6 −3.60{}^{-3.6\phantom{0}}58.1 −7.10{}^{-7.1\phantom{0}}55.2 -10.032.2 -33.0
Appendix
Table 13: Our uncorrected text encoders are competitive with corrected SlimGPT. MTEB(eng, v2) score for the nine text encoders, comparing our method without correction against SlimGPT with and without correction. Our method outperforms corrected SlimGPT at every evaluated sparsity target on BGE-B, BGE-L, Qwen3 0.6B, and Qwen3 8B, and on seven of nine encoders between 40% and 60% sparsity.
 EagerCompiled
BatchDenseA=1A=8A=64DenseA=1A=8A=64
15.402 ms1.00 ×\times0.95 ×\times1.00 ×\times3.050 ms0.98 ×\times0.97 ×\times1.00 ×\times
1 †\dagger––––1.812 ms1.09 ×\times1.25 ×\times1.26 ×\times
45.666 ms1.06 ×\times1.02 ×\times1.03 ×\times3.196 ms0.99 ×\times1.00 ×\times1.04 ×\times
165.806 ms1.01 ×\times0.99 ×\times0.97 ×\times5.226 ms1.42 ×\times1.52 ×\times1.57 ×\times
3210.461 ms1.22 ×\times1.54 ×\times1.57 ×\times9.214 ms1.59 ×\times1.67 ×\times1.72 ×\times
Appendix
Table 14: GPU-friendly tiling helps latency. Latency of dense and 60% pruned AugReg ViT-B/16 models across batch sizes, under eager execution and torch.compile , for unconstrained pruning, i.e., A=1A=1 , and pruning in blocks of 8 or 64 neurons, reported as speedup over the dense model. Under eager execution, latency is launch-bound up to batch size 16 and pruning yields no speedup, while tiling accounts for most of the gains at larger batches. Compilation lifts speedups across the board, up to 1.79×1.79\times at batch size 128, and at batch size 1 gains only materialize without launch overhead.
 ImageNet-1kTransfer Average
GFLOPsA=1A=8A=64A=1A=8A=64
31.782.682.5 -0.182.4 -0.278.978.9 +0.078.8 -0.1
28.280.179.6 -0.579.5 -0.676.976.7 -0.276.7 -0.2
24.875.775.4 -0.375.4 -0.374.173.4 -0.773.3 -0.8
21.269.169.1 +0.069.1 +0.066.869.7 +2.969.3 +2.5
17.859.358.6 -0.758.2 -1.163.763.8 +0.163.7 +0.0
Appendix
Table 15: GPU-friendly tiling can affect accuracy at high sparsity. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for pruned AugReg ViT-B/16, with unconstrained pruning, i.e., A=1A=1 , or pruning constrained to remove MLP neurons in blocks of 8 or 64. Superscripts indicate the delta relative to A=1A=1 . Tiling costs little at up to moderate sparsity, but at 60% sparsity pruning in blocks of 8 and 64 trails unconstrained pruning by 3.4 and 3.2 points on ImageNet-1k, respectively.
 EagerCompiled
GFLOPsA=1A=8A=64A=1A=8A=64
Dense (35.1)3572 samples/s4143 samples/s
31.70.75 ×\times1.03 ×\times1.06 ×\times1.03 ×\times1.07 ×\times1.08 ×\times
28.20.70 ×\times1.10 ×\times1.14 ×\times1.11 ×\times1.16 ×\times1.17 ×\times
24.80.91 ×\times1.19 ×\times1.23 ×\times1.24 ×\times1.27 ×\times1.28 ×\times
21.21.03 ×\times1.30 ×\times1.34 ×\times1.35 ×\times1.39 ×\times1.42 ×\times
Appendix
Table 16: Pruning improves throughput. Throughput of dense and pruned AugReg ViT-B/16 models in fp16 at batch size 128, under eager execution and torch.compile , for unconstrained pruning, i.e., A=1A=1 , and pruning in blocks of 8 or 64 neurons, reported as speedup over the dense model. In eager mode, ragged widths can be slower than the dense model, while pruning in blocks improves throughput by up to 1.63×1.63\times over unconstrained pruning at 20% sparsity. Compilation nearly eliminates this gap, and models pruned to 60% sparsity reach up to 1.80×1.80\times dense throughput.
MethodViT-S/16ViT-B/16ViT-L/16ViT-H/14
SparseGPT4.5min5.0min11min20min
FPTP5.6min9.0min19min35.6min
OPTIN12.5min1.0h8.0h34.6h
VBP2.6h6.6h20.6h53.7h
SnapViT2.2min4.0min11.2min28.8min
Ours8.7min13.4min24.5min40.7min
Appendix
Table 17: Our method is competitive on runtime. Wall-clock time to prune models to all six sparsity targets evaluated in this paper, on a single NVIDIA RTX A6000. Our method and SnapViT produce any-budget models in a single run, OPTIN reuses its leave-one-out search across targets, and all other methods must be run end-to-end per target.
  Sparsity
  20% / 28.2 G40% / 21.2 G60% / 14.2 G
DatasetDenseImageNetTaskImageNetTaskImageNetTask
CIFAR 1097.393.597.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.5}}82.496.1 +13.769.586.8 +17.3
CIFAR 10085.075.583.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.3}}58.178.0 +19.943.356.5 +13.2
Aircraft24.626.725.0 {}^{\phantom{0}{\color[rgb]{0.7,0,0}-1.7}}21.423.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.3}}12.516.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.2}}
EuroSAT92.692.193.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.2}}91.694.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.6}}84.490.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+6.3}}
Appendix
Table 18: Task-aligned calibration improves performance. Top-1 k-NN accuracy on transfer datasets for AugReg ViT-B/16 pruned with our method, using rankings optimized on ImageNet-1k or the target task’s training split. Task-aligned calibration improves 25 of 27 settings, gaining up to 26.6% on Flowers 102 at 40% sparsity and 20.8% on Pets at 60%.
 Sparsity
 20% / 28.2 G40% / 21.2 G60% / 14.2 G
DatasetCosineNativeCosineNativeCosineNative
ImageNet-1k79.280.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.9}}64.269.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.9}}36.345.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.9}}
CIFAR 1092.995.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.2}}81.876.2 {}^{\phantom{0}{\color[rgb]{0.7,0,0}-5.6}}66.768.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.0}}
CIFAR 10074.179.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.5}}59.653.4 {}^{\phantom{0}{\color[rgb]{0.7,0,0}-6.2}}41.343.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.7}}
Aircraft24.927.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.4}}20.123.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.0}}12.719.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+6.9}}
Appendix
Table 19: Optimizing rankings with the backbone’s native loss improves pruning performance. Top-1 k-NN accuracy for AugReg ViT-B/16 pruned with our method, with rankings optimized by cosine-distance alignment to dense features (Cosine) or with the native cross-entropy loss (Native). Column groups correspond to 20%, 40%, and 60% sparsity. The native loss gains up to 13.5% on Pets at 60% sparsity, but trails cosine by up to 6.2% on CIFAR 100 at 40%.
 Sparsity
 20% / 28.2 G40% / 21.2 G60% / 14.3 G
DatasetCosineNativeCosineNativeCosineNative
ImageNet-1k74.776.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.4}}53.561.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.6}}20.830.9 +10.1
CIFAR 1092.695.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.4}}67.471.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.2}}52.357.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.8}}
CIFAR 10072.977.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.4}}42.646.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.7}}26.731.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.5}}
Aircraft34.646.5 +11.915.123.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.2}}8.613.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.7}}
Appendix
Table 20: Optimizing rankings with the backbone’s native loss improves pruning performance. Top-1 k-NN accuracy for SigLIPv2 ViT-B/16 pruned with our method, with rankings optimized by cosine-distance alignment to dense features (Cosine) or with the native sigmoid image-text loss (Native). Column groups correspond to 20%, 40%, and 60% sparsity. The native loss gains up to 19.0% on Pets at 60% sparsity, but trails cosine by up to 2.9% on EuroSAT at 40%.
 Unique Samples
GFLOPs6412825651210002000 †\dagger400080009600
31.781.881.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.0}}82.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.3}}82.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.4}}82.5 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.7}}82.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.8}}82.9 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.1}}82.9 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.1}}83.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.2}}
28.277.477.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.2}}78.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.7}}78.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.2}}79.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.7}}80.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.7}}80.5 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.1}}81.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.6}}81.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.7}}
24.870.371.3 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.0}}71.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+1.1}}72.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.5}}74.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.8}}75.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.4}}77.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+6.8}}77.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.3}}77.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.4}}
21.259.462.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.7}}63.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+3.6}}65.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.8}}67.6 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.2}}69.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+9.7}}70.7 +11.371.1 +11.771.7 +12.3
17.846.649.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.6}}51.7 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.1}}55.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+8.5}}57.2 +10.659.3 +12.759.8 +13.260.2 +13.660.9 +14.3
Appendix
Table 21: More unique calibration samples generally improve ranking optimization. Top-1 k-NN accuracy on ImageNet-1k for AugReg ViT-B/16 pruned with our method, varying the number of unique ImageNet-1k samples available to ranking optimization under fixed compute, i.e., 300 steps, at batch size 32. Superscripts give the change relative to 64 samples, and †\dagger marks our default. Using 9600 samples gains up to 14.3 points over 64 samples at 50% sparsity, and up to 2.6 points over the default at 40%, but trails it by 2.0 points at 60%.
 Sparsity
Threshold10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
ImageNet-1k
Per-block82.579.774.365.254.040.6
Cross-block82.5 +0.080.2 +0.576.2 +1.969.6 +4.457.2 +3.245.7 +5.1
Unified82.6 +0.180.1 +0.475.7 +1.469.1 +3.959.3 +5.345.2 +4.6
Transfer Average
Appendix
Table 22: Unified ranking matches cross-block thresholding on AugReg. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for AugReg ViT-B/16 pruned with our method, with separate neuron and head rankings under per-block or cross-block thresholding, or our default unified ranking. Superscripts indicate the delta relative to per-block.
 Sparsity
Threshold10% / 41.9 G20% / 37.3 G30% / 32.5 G40% / 28.0 G50% / 23.4 G60% / 18.8 G
ImageNet-1k
Per-block81.477.869.329.915.58.0
Cross-block82.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.6}}80.5 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.7}}72.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.9}}64.8 +34.947.7 +32.232.1 +24.1
Unified81.9 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.5}}80.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+2.4}}76.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.1}}69.4 +39.556.4 +40.938.9 +30.9
Transfer Average
Appendix
Table 23: Unified ranking improves DINOv3 performance at high sparsity. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for DINOv3 ViT-B/16 pruned with our method, with separate neuron and head rankings under per-block or cross-block thresholding, or our default unified ranking. Superscripts indicate the delta relative to per-block. Cross-block thresholding gains up to 34.9% over per-block on ImageNet-1k at 40% sparsity, and unifying neurons and heads adds up to 8.7% at 50%.
 Sparsity
Threshold10% / 31.7 G20% / 28.2 G30% / 24.7 G40% / 21.2 G50% / 17.7 G60% / 14.3 G
ImageNet-1k
Per-block79.071.265.651.440.626.8
Cross-block79.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.4}}77.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+6.0}}72.0 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+6.4}}63.2 +11.848.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.8}}33.8 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+7.0}}
Unified79.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+0.4}}76.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.9}}70.4 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.8}}61.1 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+9.7}}46.2 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+5.6}}30.9 {}^{\phantom{0}{\color[rgb]{0,0.55,0}+4.1}}
Transfer Average
Appendix
Table 24: Cross-block thresholding performs best for SigLIPv2. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for SigLIPv2 ViT-B/16 pruned with our method, with separate neuron and head rankings under per-block or cross-block thresholding, or our default unified ranking. Superscripts indicate the delta relative to per-block. Unified ranking trails cross-block thresholding by up to 2.5% on the transfer average at 40% sparsity.
 Sparsity
 10%15% †\dagger20%25% †\dagger30%35% †\dagger40%45% †\dagger50%55% †\dagger60%
ImageNet-1k82.781.579.978.275.973.369.464.259.152.143.3
Transfer Average78.977.876.875.673.371.068.666.263.559.355.2
Appendix
Table 25: Our rankings interpolate to unseen sparsity targets. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for pruned AugReg ViT-B/16 with a ranking optimized over seven sparsity anchors, from 10% to 70% in 10% increments, and evaluated every 5%. Targets marked †\dagger lie between anchors. Performance degrades smoothly at the evaluated intermediate targets.
 Sparsity 
Temperature20% / 28.2 G40% / 21.2 G60% / 14.2 GMean
ImageNet-1k
No optimization78.642.517.046.1
τ\tau = 0.0179.767.741.663.0
τ\tau = 0.0580.469.744.164.8
τ\tau = 0.180.370.644.565.1
Appendix
Table 26: Low temperatures improve ranking optimization. Top-1 k-NN accuracy on ImageNet-1k and transfer datasets for AugReg ViT-B/16 pruned with our method, varying the sigmoid temperature τ\tau of the ranking optimization. The first row uses plain scores, without ranking optimization. The default setting is highlighted in orange . Low temperatures perform best, with mean accuracy peaking at τ=0.1\tau=0.1 . Mean accuracy decreases steadily above 0.20.2 and approaches the unoptimized baseline at τ=5.0\tau=5.0 .
 Sparsity
 10% / 31.7 G20% / 28.2 G30% / 24.8 G40% / 21.2 G50% / 17.8 G60% / 14.2 G
ImageNet-1k82.5±0.1482.5\pm 0.1479.9±0.1479.9\pm 0.1475.6±0.1775.6\pm 0.1769.1±0.0969.1\pm 0.0959.5±0.5459.5\pm 0.5444.5±0.6544.5\pm 0.65
Transfer Average78.7±0.2078.7\pm 0.2076.7±0.2376.7\pm 0.2373.6±0.4473.6\pm 0.4468.7±1.8168.7\pm 1.8164.1±0.5564.1\pm 0.5555.9±0.1255.9\pm 0.12
Appendix
Table 27: Performance is generally stable across random seeds. Top-1 k-NN accuracy on ImageNet-1k and the transfer average for AugReg ViT-B/16 models pruned with our method, reported as mean ±\pm sample standard deviation across three seeds. Transfer accuracy is first averaged across the nine datasets within each seed.
More in this topic

Elastic Neural Networks

See all 9 papers

Explore similar work

CardsList
Oct 10, 2026

ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer

Foundation models need to support diverse inference budgets, yet training a separate model for each budget duplicates pretraining effort. Elastic models train several nested sub-networks inside one weight-shared supernet, but existing recipes face three obstacles: variable per-step workloads that hinder distributed scaling, limited hyperparameter (HP) transfer across model sizes, and no guarantee that all sub-networks are maximally updated at once. We present \Ours, an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains every sub-network's exact hidden states as prefixes, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone. We further introduce \muE, an extension of maximal update parameterization (\mup) to \Ours and other elastic architectures. Its scaling rules provably keep all sub-networks of a supernet maximally updated simultaneously (\muI), and keep doing so when the supernet is widened or given more shells; sweeps confirm that HPs tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets (\mbox{\muT}). A single \Ours run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.

Can Jin, Yan Kang, Yifan Gong +5

Rutgers University · Adobe Research

Elastic Neural NetworksTransformerHyperparameter TransferDynamic Neural Networks
AskPDF
Aug 25, 2026

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.

Peng Liu, Huibing Zeng, Yiqun Zhang +2

School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Information and Artificial Intelligence, Yangzhou University, Yangzhou 225000, China

LLM PruningTransformer PruningNeural Network Pruning
AskPDF
Aug 9, 2026

Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers

A single model scale challenges the flexibility of a production retrieval system: some settings need it faster, others need a smaller index, and the right trade-off changes with the workload. In the context of information retrieval (IR), a transformer-based model can be made smaller in three ways---using fewer layers, passing fewer tokens through the upper layers, or producing a shorter embedding---and each way saves a different compute resource. These options have been studied one at a time, each as its own method with its own code and training setup, which makes them hard to combine or adapt to a new model. We present~\ours to bring all three under one simple abstraction: a single object names any size the model can run at, and a short schedule lists the sizes to train. Training then produces one checkpoint that serves all of those sizes, and at deployment the user picks any of them. The same abstraction covers both retrievers and rerankers and both encoder and decoder models, as it works through interfaces that Hugging Face transformers already expose; a new backbone is a configuration change, not new modeling code. Prior methods---Matryoshka embeddings, early exit, 2DMatryoshka (e.g., Starbucks), and layerwise token compression---become special cases of our unified abstraction. The same interface also enables MatryoshkaLTC (MLTC), which jointly trains several token-compression ratios in one retriever checkpoint. To validate our framework, we train 20 checkpoints across three backbones and two tasks: the quality curves are smooth, one checkpoint costs little over a model trained for a single size, and a controlled study confirms the wallclock speedups. We release the framework and all checkpoints as a resource for building elastic retrieval systems.

Yu Wang, Shengyao Zhuang, Xueguang Ma +4

University of Utah · 2The University of Queensland · University of Waterloo +1

Model CompressionDynamic Neural NetworksElastic Neural NetworksInformation RetrievalToken Compression
AskPDF