Dynamic sparse training (DST) aims to reduce training costs while maintaining accuracy. Existing DST methods mask out parameters but keep matrix dimensions unchanged. Training speedups therefore depend on specialized support, such as sparse kernels. We introduce BARTER, a structured DST framework that reduces training cost using standard dense kernels by keeping training tensors physically compact while dynamically reallocating capacity across layers. Unlike existing works that require computing gradients for all parameters, BARTER instead uses statistics collected only from active units during ordinary backpropagation to prune and clone attention heads and feed-forward channels under separate fixed global budgets. At each topology update, it resizes the affected parameter tensors and remaps their optimizer states, so subsequent training steps operate directly on the compact model. Training and fine-tuning experiments with Transformer-based vision models on two image classification benchmarks at densities from 0.1 to 0.7 show competitive Top-1 accuracy relative to full-tensor DST baselines. On an NVIDIA L40, BARTER achieves speedups of up to 6.29× in measured training segments and reduces peak allocated GPU memory by up to 81.3% relative to these implementations. An iPhone 13 Pro Max training benchmark additionally shows a 2.78× speedup over the full-tensor DST baseline.
Figures & tables
Figure 1: Full-size masked training versus compact dynamic training with BARTER . The plot compares CCT/CIFAR-100 accuracy and schedule-weighted training step-time estimates for RigL and BARTER under dense-kernel execution. Labels indicate density.
Figure 2: Overview of our approach, BARTER . Budgeted initialization produces a compact model with fewer attention heads and FFN channels while retaining all Transformer layers. During training, statistics from active units guide capacity reallocation across layers. For each unit type, the method selects the eligible receiving layer with the highest growth priority, prunes eligible units with the lowest pruning scores outside that layer, and adds the same number of units by cloning parents with the highest growth scores within it. These updates preserve the global head and FFN budgets separately. Each clone copies its parent’s input parameters, while the parent’s output weights are split between the parent and clone. The affected parameter tensors and optimizer states are resized accordingly, and training continues on the compact model with standard dense kernels.
(a) CIFAR-100 (CCT)
Dense ( ρ=1 ): Acc 73.49 ±0.10 , F1 73.29 ±0.11
ρ=0.1
ρ=0.3
ρ=0.5
ρ=0.7
Method
Acc ↑
F1 ↑
Acc ↑
F1 ↑
Acc ↑
F1 ↑
Acc ↑
F1 ↑
SET
63.21 ±0.55
62.80 ±0.56
70.26 ±0.30
70.08 ±0.31
72.25 ±0.06
72.04 ±0.10
73.21 ±0.16
72.98 ±0.13
SNFS
63.29 ±0.36
62.87 ±0.35
69.45 ±0.14
69.31 ±0.14
70.92 ±0.53
70.74 ±0.51
72.17 ±0.17
71.97 ±0.12
RigL
67.01 ±0.17
66.67 ±0.08
71.74 ±0.14
71.57 ±0.10
72.62 ±0.21
72.40 ±0.22
73.29 ±0.08
73.09 ±0.07
Table 1: Classification performance (accuracy/macro-F1) of models trained from scratch. Best sparse results per task and density in bold and orange ; second-best in underlined italics . Mean ± std (%) over three seeds.
Figure 3: Accuracy, GPU training time per epoch, and peak allocated GPU memory for CCT on CIFAR-100 and ViT on Tiny-ImageNet. Colors distinguish methods. Each sparse-method curve connects four points corresponding to densities ρ=0.1,0.3,0.5,0.7 . Dense is a single full-density reference ( ρ=1 ). Error bars show sample standard deviations.
Figure 4: Estimated training GFLOPs per image versus exported parameter counts for (a) CCT and (b) ViT. Points on each colored curve correspond to densities ρ=0.1,0.3,0.5,0.7 from left to right. The star marks the overlapping results of Dense, SET, SNFS, RigL, and SRigL. S 2 ViTE and BARTER have identical exported parameter counts at each density.
Figure 5: Fine-tuning on CIFAR-100. (a) Results after 100 epochs at sparse densities ρ=0.1,0.3,0.5,0.7 . (b) BARTER ’s 100-epoch (filled circles) and compute-matched (hollow circles) results, compared with RigL at the same densities. Numeric labels indicate density; Dense is a full-density reference. Error bars show sample standard deviations.
Training time
Method
ms/step
s/epoch
Dense
4814.77
1726.77
RigL
4927.10
1699.36
BARTER
1804.57
610.33
Table 2: CCT training on iPhone 13 Pro Max. RigL and BARTER use ρ=0.3 .
Figure 6: Ablations of capacity reallocation and selection on CCT/CIFAR-100 at ρ=0.5 . Bars show ACC and macro-F1 differences (variant minus BARTER ) in percentage points, with variant scores (%) in parentheses. Values are means over three seeds per method.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Model
ρ
Heads H
FFN channels C
KH
KC
CCT
0.1
11 (1–2)
371 (53)
1
21
0.3
34 (4–5)
1,067 (152–153)
2
60
0.5
56 (8)
1,795 (256–257)
3
101
0.7
78 (11–12)
2,524 (360–361)
5
142
ViT
0.1
11 (1–2)
359 (39–40)
1
20
0.3
33 (3–4)
1,033 (114–115)
2
58
Appendix
Table 3: Global capacity budgets and reference transfer counts for the main experiments. Initial per-layer counts are shown in parentheses. All scheduled head and FFN-channel transfers, KH and KC , were executed in these runs.
Setting
CCT
ViT
Input resolution
32×32
64×64
Transformer layers
7
9
Embedding dimension
256
192
Heads per layer
16
12
Head dimension
16
16
FFN hidden dimension
512
384
Appendix
Table 4: Full-density model configurations. Fine-tuning retains the ViT backbone.
Setting
CCT
ViT
ViT fine-tuning
Epochs
200
100
100
Batch size
128
256
256
Steps per epoch
351
1,054
527
Peak learning rate
3×10−4
2×10−3
10−3
Minimum learning rate
0
10−6
10−6
Weight decay
0.05
0.05
0.05
Appendix
Table 5: Training settings. Fine-tuning epochs refer to the initial phase; warmup is a fraction of optimizer steps.
Figure 7: Schedule-weighted training step-time estimates for CCT on CIFAR-100 and ViT on Tiny-ImageNet. Error bars show sample standard deviations.
Method
ρ=0.1
ρ=0.3
ρ=0.5
ρ=0.7
Measured training time (s/epoch)
Dense
68.45±0.01
68.45±0.01
68.45±0.01
68.45±0.01
SET
69.12±0.04
69.05±0.03
69.11±0.02
69.18±0.05
SNFS
68.73±0.03
68.65±0.03
68.72±0.05
68.80±0.07
RigL
68.97±0.05
69.05±0.10
69.09±0.02
69.24±0.05
SRigL
69.27±0.05
69.36±0.07
69.90±0.02
70.78±0.22
Appendix
Table 6: GPU training efficiency for CCT on CIFAR-100. Values are mean ± sample standard deviation. Dense is a full-density reference repeated across columns.
Method
ρ=0.1
ρ=0.3
ρ=0.5
ρ=0.7
Estimated training GFLOPs/image
Dense
7.089840
7.089840
7.089840
7.089840
SET
7.089840
7.089840
7.089840
7.089840
SNFS
7.089840
7.089840
7.089840
7.089840
RigL
7.089840
7.089840
7.089840
7.089840
SRigL
7.089840
7.089840
7.089840
7.089840
Appendix
Table 7: Estimated training computation and exported model size for CCT on CIFAR-100. All sample standard deviations are zero at the reported precision. Dense is a full-density reference repeated across columns.
Method
ρ=0.1
ρ=0.3
ρ=0.5
ρ=0.7
Measured training time (s/epoch)
Dense
39.78±0.48
39.78±0.48
39.78±0.48
39.78±0.48
SET
44.50±0.92
44.32±0.43
44.35±0.34
44.76±0.51
SNFS
42.31±0.19
42.44±0.40
42.64±0.31
42.89±0.26
RigL
44.03±0.25
44.46±0.29
44.46±0.54
44.58±0.45
SRigL
45.04±0.39
45.19±0.63
46.13±0.49
47.32±0.40
Appendix
Table 8: GPU training efficiency for ViT on Tiny-ImageNet. Values are mean ± sample standard deviation. Dense is a full-density reference repeated across columns.
Method
ρ=0.1
ρ=0.3
ρ=0.5
ρ=0.7
Estimated training GFLOPs/image
Dense
1.137138
1.137138
1.137138
1.137138
SET
1.137138
1.137138
1.137138
1.137138
SNFS
1.137138
1.137138
1.137138
1.137138
RigL
1.137138
1.137138
1.137138
1.137138
SRigL
1.137138
1.137138
1.137138
1.137138
Appendix
Table 9: Estimated training computation and exported model size for ViT on Tiny-ImageNet. All sample standard deviations are zero at the reported precision. Dense is a full-density reference repeated across columns.
Method
Density
ACC (%)
F1 (%)
PFLOPs
Dense
1.0
72.23±0.42
72.01±0.45
15.27612
SET
0.1
58.81±0.42
58.26±0.48
15.27612
SNFS
0.1
58.89±0.26
58.25±0.29
15.27612
RigL
0.1
63.19±0.53
62.80±0.56
15.27612
SRigL
0.1
60.70±1.51
60.27±1.55
15.27612
S 2 ViTE
0.1
59.66±0.74
59.01±0.74
15.27612
Appendix
Table 10: Fine-tuning results after 100 epochs. ACC and macro-F1 are percentages (mean ± sample standard deviation over three seeds). Compute is cumulative forward-plus-backward training PFLOPs.
Method
Density
Epochs
PFLOPs
ACC (%)
F1 (%)
RigL
0.1
100.00
15.27612
63.19±0.53
62.80±0.56
BARTER
0.1
100.00
1.68580
61.21±0.12
60.68±0.17
BARTER
0.1
906.17
15.27616
65.65±0.59
65.33±0.62
RigL
0.3
100.00
15.27612
68.38±0.31
68.10±0.34
BARTER
0.3
100.00
4.71074
69.02±0.66
68.81±0.72
BARTER
0.3
324.28
15.27618
70.39±0.98
70.18±0.95
Appendix
Table 11: Fine-tuning with additional compute-matched training. RigL is the strongest sparse baseline at both densities. ACC and macro-F1 are percentages (mean ± sample standard deviation).
Method
ACC
F1
Static
71.80±0.14
71.59±0.16
Random receiver
72.49±0.29
72.26±0.30
Random donor
71.54±0.75
71.32±0.75
Random pair
71.20±0.65
70.98±0.65
BARTER
72.85±0.16
72.64±0.14
Appendix
Table 12: Ablations on CCT/CIFAR-100 at ρ=0.5 . ACC and macro-F1 (%) are reported as mean ± sample standard deviation over three seeds per method. Best means are bold and orange .
Static
BARTER
ρ
ACC
Macro-F1
ACC
Macro-F1
CCT/CIFAR-100
0.1
63.16±0.27
62.73±0.25
66.13±0.40
65.79±0.44
0.3
70.43±0.13
70.27±0.15
71.38±0.36
71.21±0.32
0.5
71.80±0.14
71.59±0.16
72.85±0.16
72.64±0.14
0.7
72.99±0.46
72.75±0.48
73.26±0.28
73.05±0.28
Appendix
Table 13: Static compact training versus BARTER on the two from-scratch tasks. ACC and macro-F1 (%) are mean ± sample standard deviation over three seeds per method. The higher mean in each comparison is bold and orange .
ρ
Method
Training time (s/segment)
Step time (ms/step)
Peak memory (GiB)
CCT/CIFAR-100
0.1
Static
11.53±0.03
32.69±0.05
2.222±0.000
0.1
BARTER
11.90±0.06
33.27±0.21
2.197±0.034
0.3
Static
25.18±0.03
71.49±0.02
4.263±0.001
0.3
BARTER
25.48±0.02
71.71±0.20
4.280±0.011
0.5
Static
36.90±0.08
104.92±0.29
6.296±0.000
Appendix
Table 14: GPU training efficiency of Static and BARTER . Training time is measured over an epoch-length segment; step time is a schedule-weighted estimate. Values are mean ± sample standard deviation. Both methods use the same hardware, batch size, segment length, and measurement protocol.
Training GFLOPs/image
Parameters (M)
ρ
Static
BARTER
Static
BARTER
CCT/CIFAR-100
0.1
0.750
0.750
0.4802
0.4802
0.3
2.166
2.166
1.2141
1.2141
0.5
3.569
3.569
1.9480
1.9480
0.7
4.973
4.973
2.6824
2.6824
Appendix
Table 15: Estimated training compute and exported model size. Training GFLOPs per image are rounded to three decimals; parameter counts use four decimals. Reported standard deviations are zero at these precisions. Static parameter counts use last.pt ; BARTER counts use the validation-selected best.pt .
Multiplier
Head transfers
FFN transfers
ACC (%)
Macro-F1 (%)
1× (reference)
3
98
52.77±0.23
52.34±0.29
2×
6
196
53.15±0.89
52.69±0.94
4×
12
392
53.30±0.60
52.92±0.66
6×
18
588
53.49±0.48
53.11±0.59
Appendix
Table 16: Effect of increasing capacity-transfer counts on ViT/Tiny-ImageNet at ρ=0.5 . ACC and macro-F1 are mean ± sample standard deviation over seeds 1, 2, and 4. Executed transfer counts and classification scores are taken from the same validation-selected checkpoints. Best means are bold and orange .
Incremental topology-update cost
Multiplier
ms/step
s/schedule
1× (reference)
0.019±0.001
1.989±0.069
2×
0.038±0.001
4.051±0.080
4×
0.077±0.003
8.067±0.362
6×
0.117±0.005
12.321±0.494
Appendix
Table 17: Estimated incremental topology-update cost for the settings in Table 16 . Values are mean ± sample standard deviation across three seeds. Per-step costs are averaged over all optimizer steps; cumulative costs cover the 100-epoch schedule.
Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.
Mohammed Adnan, Rohan Jain, Tom Jacobs +4
University of Calgary · 2Vector Institute · 3CISPA Helmholtz +2
Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.
Qiao Xiao, Boqian Wu, Patrik Okanovic +6
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available here.