AC/DC on a Budget - Alternating Sparse Phases

Rahul Nittala; Advait Gadhikar; Tom Jacobs; Rebekka Burkholz

cs.LGOct 20, 2026

AC/DC on a Budget - Alternating Sparse Phases

Authors: Rahul Nittala, Advait Gadhikar, Tom Jacobs, Rebekka Burkholz

Organizations: CISPA Helmholtz Center for Information Security, Germany

Abstract

State-of-the-art neural network sparsification relies on alternating dense and sparse training phases. While computationally expensive, dense phases are critical for performance. Our empirical analysis reveals the mechanics behind this gain: upon reintroduction, previously masked parameters receive disproportionately large gradients that drive essential mask rewiring and parameter sign-flipping. Building on these insights, we optimize the performance-to-FLOPs Pareto frontier through three innovations: (1) zero-initializing masked parameters during dense phases to promote relevant sign flips, (2) shortening dense phases, and (3) replacing the dense phase entirely with a sparse rewiring phase that updates only the active mask and a small, high-gradient fraction of out-of-mask weights. Evaluated on CIFAR100 and ImageNet, our variant ARC (Alternating Rewiring Compression) matches baseline AC/DC performance at a fraction of the computational cost, establishing a superior performance-to-FLOPs frontier also compared to standard sparse-to-sparse algorithms. Experiments with GPT-2 on OpenWebText further show improvements in validation perplexity and average zero-shot performance

Figures & tables

Figure 1: Gradient–weight dynamics across AC/DC phases. Gradient versus weight scatter plots for layer4.0.conv2 of a 90% sparse ResNet18 on CIFAR100. Top and bottom rows show the beginning and end of each phase, respectively. During sparse phases, many active weights move toward small magnitude and small gradient. During dense phases, many reintroduced out-of-mask weights remain near zero but receive large gradients.

Panel of Figure 2 — Figure 2: Timing of final sign alignment. For each parameter retained in the final sparse mask, we record the phase in which it flips sign for the last time. The y-axis is normalized by the final mask size. DW, D, S, and SC denote dense warmup, dense, sparse, and sparse cooldown phases for AC/DC; in ARC, the corresponding non-sparse phases are rewiring phases on $A_{t}=S_{t}\cup G_{t}$ . (a) In AC/DC, most final-mask parameters align during dense warmup, while later sign corrections occur mainly during dense phases. (b) The analogous ARC diagnostic is discussed after ARC is introduced in Section 4 .

Panel of Figure 3 — Figure 3: Sparse phases preserve activation subspaces. Top- $r$ activation Gram subspace similarity for layer3.1.conv2 of a 90% sparse ResNet18 on CIFAR100. We show an early and later AC/DC cycle. Sparse phases maintain higher subspace similarity. Dense phases show larger but smooth drift.

Figure 4: Reducing and sparsifying rewiring phases. Results for ResNet18 on CIFAR100. Left: varying the number of dense epochs per AC/DC cycle at 90% sparsity. Right: comparing AC/DC with ARC variants using different

G_{t}

budgets across target sparsities.

Figure 5: Performance versus theoretical FLOPs. Accuracy–FLOP tradeoff for AC/DC, ARC variants, and RigL on CIFAR100. The horizontal axis reports epoch-averaged theoretical training GFLOPs per image, including support-selection probes.

	Sparsity=0.95	Sparsity=0.9	Sparsity=0.8
ARC (0.1)	$73.91\pm 0.21$	$74.45\pm 0.14$	$74.94\pm 0.21$
ARC (0.1) + Zero init	$73.94\pm 0.26$	$74.74\pm 0.36$	$75.42\pm 0.19$
ARC (0.3) + Zero init	75.17 $\pm$ 0.98	75.21 $\pm$ 0.26	75.54 $\pm$ 0.58
ARC (0.1) + Zero init 5-D, 15-S	$73.76\pm 0.34$	$74.44\pm 0.06$	$75.16\pm 0.20$
AC/DC Baseline	$73.77\pm 0.17$	$74.70\pm 0.40$	$75.08\pm 0.30$
AC/DC + Zero init	74.86 $\pm$ 0.38	75.07 $\pm$ 0.38	75.88 $\pm$ 0.20

Table 1: AC/DC, ARC and RigL on CIFAR100 with ResNet18.

	Sparsity=0.7		Sparsity=0.5
	Test Acc	GFLOPs	Test Acc	GFLOPs
ARC (0.2)	76.50 $\pm$ 0.21	13.34	77.82 $\pm$ 0.19	18.11
ARC (0.2) + Zero init	$76.21\pm 0.22$	13.34	$77.67\pm 0.76$	18.11
ARC (0.2) + Zero init 10-D, 20-S	$76.20\pm 0.08$	12.59	$77.40\pm 0.27$	17.36
AC/DC Baseline	$75.89\pm 0.31$	19.01	$77.433\pm 0.30$	21.52
AC/DC + Zero init	75.891 $\pm$ 0.74	19.01	77.82 $\pm$ 0.12	21.52

Table 2: AC/DC, ARC and RigL on ImageNet with DeiT-Small. We report top-1 accuracy (%) and epoch-averaged theoretical training GFLOPs per image, including support-selection probes.

Published baselines
Method	Top-1 (%)	Inference FLOPs	Training FLOPs
RigL $1\times$ (ERK)	$69.70\pm 0.17$	$0.12\times$	$0.13\times$
Top-KAST	$71.96$	$0.08\times$	$0.22\times$
AC/DC	$73.14\pm 0.20$	$0.11\times$	$0.53\times$
Our runs
ARC–Stale	$72.083\pm 0.077$	$0.0920\times$	$0.2321\times$

Table 3: ResNet50/ImageNet at 95% sparsity. Published baselines from Table 2 of ( Peste et al., 2021 ) and our three-seed ARC results. Inference FLOPs per image and full-run training FLOPs are normalized to dense; ARC includes selection probes. Top-KAST uses 95% forward and 50% backward sparsity.

Method	Validation loss $\downarrow$	Perplexity $\downarrow$	Zero-shot average $\uparrow$
AC/DC	$3.13690$	$23.0327$	$36.27$
RigL	$3.13847$	$23.0685$	$35.67$
ARC–Stale	$3.06056$	$21.3395$	$36.35$
ARC–Zero	$\mathbf{3.05646}$	$\mathbf{21.2521}$	$\mathbf{37.29}$

Table 4: GPT-2 124M on OpenWebText at 80% target sparsity. Results use one training seed (1337) and the end-of-training checkpoint. The zero-shot average is the arithmetic mean of seven task scores (%). Bold indicates the best result among the compared sparse methods.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: AC/DC & ARC Schematic. AC/DC finds a sparse network, by starting with a dense warmup, then alternating between dense and sparse phases and finally a sparse finetuning phase. ARC replaces the alternating dense phases with sparse rewiring phases.

Configuration	CIFAR100 Experiment	ImageNet Experiment
Dataset	CIFAR100	ImageNet
Model	ResNet18	DeiT-small
LR	$0.1$	$0.05$
WD	$1e-4$	$1e-1$
Epochs	$150$	$300$
Batch Size	$128$	$256$

Appendix

Table 5: Default training hyperparameters for ResNet18/CIFAR100 and DeiT-Small/ImageNet.

Configuration	CIFAR100 Experiment	ImageNet Experiment
ACDC/ARC Dense warmup epochs	15	15
ACDC/ARC Dense epochs	10	15
ACDC/ARC Sparse epochs	10	15
ACDC/ARC Sparse cooldown epochs	15	15
ARC - Batches to compute $G_{t}$	5	5
RigL - Dense gradient accumulation steps	100	10

Appendix

Table 6: Default AC/DC, ARC, and RigL settings for ResNet18/CIFAR100 and DeiT-Small/ImageNet. Shortened-phase configurations are specified separately in the corresponding experiments.

Panel of Figure 7 — Figure 7: Gradient vs parameter scatter plots of parameters of layers of a 90% sparse ResNet18 model on CIFAR100. The plots are shown for the 3rd and 4th dense and sparse phases of the 10-Dense, 10-Sparse schedule of ACDC with Stale Init . (top) plots are at the beginning of a phase and (bottom) plots are at the end of a phase.

Panel of Figure 8 — Figure 8: Gradient vs parameter scatter plots of parameters of layers of a 90% sparse ResNet18 model on CIFAR100. The plots are shown for the 3rd and 4th dense and sparse phases of the 3-Dense, 17-Sparse schedule of ACDC with Stale Init . (top) plots are at the beginning of a phase and (bottom) plots are at the end of a phase.

Figure 9: Replacing dense phases in AC/DC with updating only a random subset of parameters for CIFAR100 trained on ResNet18.

Task	AC/DC	ARC–Stale	ARC–Zero	RigL	Dense GPT-2 reference
OpenBookQA [ 48 ]	26.80	27.00	26.20	25.40	26.40
ARC-Challenge [ 9 ]	23.21	23.38	23.55	22.10	22.10
ARC-Easy [ 9 ]	38.05	37.63	39.23	37.24	40.24
WinoGrande [ 64 ]	50.28	48.86	52.64	50.05	51.85
PIQA [ 4 ]	59.09	59.90	61.59	60.34	62.40
HellaSwag [ 76 ]	29.24	29.67	29.76	28.75	31.50

Appendix

Table 7: Zero-shot evaluation of GPT-2 124M. Scores are percentages from one training seed (1337). Bold indicates the best score among the four sparse methods. The final column evaluates the released dense OpenAI GPT-2 checkpoint as an external reference.

Method	Top-1 accuracy (%)
AC/DC–Stale	$76.29\pm 0.06$
ARC–Stale	$75.96\pm 0.14$
AC/DC–Zero	$76.41\pm 0.07$
ARC–Zero	$76.06\pm 0.07$

Appendix

Table 8: 2:4 semi-structured ResNet50 on ImageNet. Top-1 accuracy (%) is reported as the mean and 95% normal-approximation confidence interval over three seeds. ARC uses 75% density in eligible layers during rewiring; AC/DC uses fully dense decompression.

Panel of Figure 10 — Figure 10: Activation subspace similarity across the first four AC/DC cycles. For layer3.1.conv2 of a 90% sparse ResNet18 trained on CIFAR100, we compare the subspaces spanned by the top $r\in\{2,3,5,10\}$ activation Gram eigenvectors. We report consecutive-checkpoint similarity $S_{r}(t-1,t)$ and similarity to the first checkpoint of each phase $S_{r}(0,t)$ . Sparse phases consistently preserve leading activation subspaces more strongly than dense phases, especially for $r=5$ and $r=10$ . Dense phases induce larger but smooth drift, consistent with their role as rewiring windows.

Panel of Figure 11 — Figure 11: Activation eigendirection similarities in the first AC/DC cycle. We compare activation-Gram eigenvectors for layer3.1.conv2 of a 90% sparse ResNet18 on CIFAR100 during the sparse phase (epochs 24–34) and dense phase (epochs 34–44). Within each panel, the top row compares consecutive checkpoints and the bottom row compares checkpoints with the phase start. Rows and columns index eigenvectors at the earlier and later checkpoints, respectively.

Panel of Figure 12 — Figure 12: Activation eigendirection similarities in the fourth AC/DC cycle. We compare activation-Gram eigenvectors for layer3.1.conv2 of a 90% sparse ResNet18 on CIFAR100 during the sparse phase (epochs 84–94) and dense phase (epochs 94–104). Within each panel, the top row compares consecutive checkpoints and the bottom row compares checkpoints with the phase start. Rows and columns index eigenvectors at the earlier and later checkpoints, respectively.

Panel of Figure 13 — Figure 13: The plot shows the fraction of weights of ResNet18 that flip their sign for the last time during a specific phase of AC/DC (3-dense, 17-sparse). Panels compare sign convergence for parameters in (a) the final converged mask and (b) the stable core weights found in the intersection of all masks.

Panel of Figure 14 — Figure 14: The plot shows the fraction of weights of specific layers of ResNet18 that flip their sign for the last time during a specific phase of AC/DC (10-dense, 10-sparse). Panels compare sign convergence for parameters of layers layer2.0.conv2 (a) and layer4.0.conv2 in the final converged mask.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 218 papers

Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU
Oct 2, 2026Jafar Badour, Maurice van Keulen, Elena MocanuDynamic Sparse Training Sparse Neural Networks
University of Twente
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
Oct 13, 2026Mehdi Makni, Natalia Ponomareva, Rahul MazumderLLM Pruning Neural Network Pruning
Operations Research Center, Massachusetts Institute of Technology · Google Research
Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural Networks Neural Network Pruning
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

Oct 2, 2026J/K move · Enter open · S save

Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

Jafar Badour, Maurice van Keulen, Elena Mocanu

University of Twente

Dynamic Sparse Training Sparse Neural Networks Neural Network Hardware Acceleration Efficient Neural Network Inference GPU Kernel Optimization Sparse Matrix Multiplication+1

Open paper AskPDF arXiv

Deep neural networks continue to grow in parameter count, driving up training and inference cost on GPUs. Sparse neural networks and Dynamic Sparse Training (DST) promise to reduce these costs, but most implementations rely on binary masks over dense tensors and recover little of the theoretical compute, memory, or energy savings. We propose SNACK, a truly sparse GPU layer that stores and computes only non-zero connections. SNACK exposes a simple PyTorch API for restructuring connections and backpropagating gradients entirely in the sparse paradigm, and ships SNACK-COO, a custom COO-format SpMM CUDA kernel with a batch-to-Streaming-Multiprocessor mapping tuned for the small-batch, high-sparsity regime typical of large-model training and single-stream inference. At the kernel level, SNACK is up to 7x faster than the masked dense baseline (Dense+Mask) and competitive with cuSPARSE, Sputnik, and FlashSparse at 95% sparsity. At 90% sparsity, a single SNACK layer accelerates training by 8x and 3.7x, and inference by 4x and 2x, over Dense+Mask and fully dense layers, respectively, while using 72% less memory than dense and substantially less energy. End-to-end, SNACK reduces GPT-2 peak training memory by up to 40% and graph-style inference latency by 4.8x over Dense+Mask at 99% sparsity.

Abstract

Neural Network Compression

Explore similar work

Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning