Compact Circulant Layers with Spectral Priors

Joseph Margaryan; Thomas Hamelryck

cs.LGFeb 25, 2026

Compact Circulant Layers with Spectral Priors

Authors: Joseph Margaryan, Thomas Hamelryck

Organizations: Department of Computer Science (DIKU), University of Copenhagen, Copenhagen, Denmark · Department of Biology, University of Copenhagen, Copenhagen, Denmark

Abstract

Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.

Figures & tables

Aspect	This work
Parameterization	Circulant/BCCB layers parameterized directly in the nonredundant RFFT half-spectrum/half-plane. As a consequence, we keep only independent Fourier coefficients (the remainder are fixed by Hermitian symmetry), avoid per-step kernel FFTs, and implement band-limits/decays via simple masks or frequency-dependent scalings.
Prior & inference	A diagonal proper complex-Gaussian prior on these spectral coordinates. As a consequence, it induces an exact stationary GP prior over discrete filters, and it pairs naturally with a Hermitian-aware low-rank + diagonal Gaussian variational posterior in effective real coordinates with closed-form KL terms.
Geometry & use cases	For FFT-diagonalizable circulant/BCCB operators, layer spectral norms are available in closed form. As a consequence, we can compute inexpensive global Lipschitz upper bounds and margin-based $\ell_{2}$ certificates as geometric diagnostics (not robustness guarantees), and we evaluate these layers as compact replacements for square projections in ViT-style architectures.

Table 1: What is new compared to prior work and why it matters . Directly parameterizing the nonredundant RFFT coefficients aligns the prior/posterior with the trainable parameters in code (enabling simple, closed-form variational terms) and makes exact spectral norms and Lipschitz diagnostics essentially free for circulant/BCCB layers.

Model	Params	Acc	Brier	ECE	AUROC
Spectral BCCB	8.6k	0.92	0.12	0.02	0.81
Spectral Circ.	8.6k	0.92	0.12	0.02	0.83
Conv2D	62.8k	0.96	0.08	0.04	0.62
Dense	622k	0.97	0.05	0.02	0.84

Table 2: MNIST (ID) and Fashion-MNIST (OOD) with SVI BNNs. Params are trainable weights (excluding biases). AUROC is based on predictive entropy; full metrics are in App. F.4 .

Panel of Figure 1 — Figure 1: Predictive-entropy KDEs for MNIST (ID; blue ) versus Fashion-MNIST (OOD; orange ), computed from the SVI posterior predictive. Left: Spectral BCCB (ours). Right: Conv2D baseline. In each image, the left panel shows the full entropy range and the right panel shows the same zoom window $H\in[0,0.12]$ nats.

Model	Acc	NLL	ECE	AUROC	FPR@95
Dense-32	0.971	0.098	0.009	0.70	0.57
Dense-128	0.971	0.118	0.012	0.71	0.56
Spectral Circ. (full)	0.971	0.136	0.017	0.68	0.60
Circulant (spatial)	0.965	0.433	0.026	0.64	0.63

Table 3: Frozen CIFAR-10 features

\rightarrow

SVI heads. We report CIFAR-10 ID accuracy/NLL/ECE and CIFAR-10-C mean AUROC and FPR@95%TPR over corruptions and severities. Higher is better for Acc/AUROC; lower is better for NLL/ECE/FPR@95. Best values in each column are in bold.

Dataset	Model	Params	Acc	NLL
CIFAR-10	Dense ViT	1,205,898	0.685	0.97
CIFAR-10	Spectral ViT	811,218	0.716	0.88
TinyImg	Dense ViT	1,248,840	0.307	3.06
TinyImg	Spectral ViT	854,160	0.344	2.94

Table 4: Dense vs spectral-circulant ViT. CIFAR-10 numbers are from the main experiment; Tiny ImageNet numbers are provided for additional context.

Figure 2: Train and validation cross-entropy over epochs for dense vs spectral ViT on Tiny ImageNet. Vertical lines mark best-validation checkpoints.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathbb{R},\mathbb{C}$	Real and complex numbers
$n,m$	Scalars (real or integer indices)
$\mathbf{x}\in\mathbb{R}^{d}$	Column vector (e.g., input signal)
$\mathbf{w}\in\mathbb{R}^{d}$	Filter vector (spatial domain)
$\mathbf{y},\mathbf{z}$	Output or latent vectors
$\mathbf{X}\in\mathbb{R}^{C_{\mathrm{in}}\times H\times W}$	Input tensor (channels $\times$ height $\times$ width)

Appendix

Table 5: Selected notation used in the main text and appendix.

Model	Weights	Biases	Total
SpectralCirculant	8,624	11	8,635
SpectralBCCB ( $C_{\text{in}}{=}C_{\text{out}}{=}1$ )	8,624	11	8,635
BCCB (spatial)	8,624	11	8,635
Conv2D ( $C_{\text{out}}{=}8$ , $k{=}3$ )	62,792	18	62,810
Dense ( $D\!\to\!D\!\to\!10$ )	622,496	794	623,290

Appendix

Table 6: Parameter counts on MNIST (

D{=}784

Model	Acc.	Brier	ECE	MCE	AUROC	FPR@95
SpectralBCCB	$0.919\pm 0.003$	$0.124\pm 0.005$	$0.016\pm 0.005$	$0.179\pm 0.081$	$0.8112\pm 0.0245$	$0.6156\pm 0.0328$
Conv2D	$0.962\pm 0.002$	$0.076\pm 0.005$	$0.038\pm 0.003$	$0.573\pm 0.069$	$0.6205\pm 0.0545$	$0.7220\pm 0.1163$
BCCB	$0.899\pm 0.003$	$0.182\pm 0.007$	$0.083\pm 0.004$	$0.373\pm 0.008$	$0.7296\pm 0.0347$	$0.8436\pm 0.0178$
SpectralCirculant	$0.921\pm 0.005$	$0.120\pm 0.005$	$0.022\pm 0.003$	$0.195\pm 0.060$	$0.8293\pm 0.0242$	$0.6369\pm 0.0296$
Circulant	$0.903\pm 0.006$	$0.175\pm 0.012$	$0.081\pm 0.005$	$0.426\pm 0.036$	$0.7811\pm 0.0190$	$0.8128\pm 0.0091$
Dense	$0.974\pm 0.001$	$0.048\pm 0.001$	$0.024\pm 0.001$	$0.623\pm 0.028$	$0.8397\pm 0.0102$	$0.5023\pm 0.0395$

Appendix

Table 7: MNIST (ID) & Fashion-MNIST (OOD). Mean

\pm

std over 3 seeds. OOD metrics from predictive entropy.

Panel of Figure 3 — Figure 3: Training dynamics with log-scaled $y$ -axis. Negative ELBO / loss for MNIST (1D and 2D models). SpectralBCCB stabilizes earlier in all settings.

Model	Acc	NLL	ECE	mean AUROC	AUROC S5	FPR@95 S5
Dense-32 (SVI)	0.971	0.098	0.009	0.6986	0.7819	0.5727
Dense-128 (SVI)	0.971	0.118	0.012	0.7076	0.7946	0.5648
RFFT (SVI) – Full	0.971	0.136	0.017	0.6816	0.7165	0.5966
Circulant (SVI)	0.965	0.433	0.026	0.6400	0.7182	0.6323

Appendix

Table 8: Frozen CIFAR-10 features

\rightarrow

SVI heads on CIFAR-10 (ID) and CIFAR-10-C (OOD). AUROC/FPR are averaged over severities S1–S5; severity 5 (S5) is shown separately.

Figure 4: Predictive-entropy KDEs on CIFAR-10 (ID, blue) vs CIFAR-10-C (OOD, severity 5, orange) for frozen-feature heads.

$K$	Weights	Comp. vs Dense-32 ( $\times$ )	Acc	mean AUROC	FPR@95 (S5)
$d/2{+}1=1025$	22,528	2.92	0.971	0.6816	0.5966
768	22,015	2.99	0.967	0.6846	0.6102
512	21,503	3.06	0.967	0.6710	0.6264
256	20,991	3.14	0.967	0.6664	0.6300
128	20,735	3.18	0.965	0.6650	0.6447
64	20,607	3.20	0.964	0.6615	0.6512

Appendix

Table 9: RFFT head ablation on frozen CIFAR-10 features (CIFAR-10 ID, CIFAR-10-C OOD). Parameter counts include spectral head

+

linear classifier (weights only). Compression is relative to Dense-32 (65,856 weights).

Panel of Figure 5 — Figure 5: Training dynamics of the global Lipschitz upper bound $\widehat{\operatorname{Lip}}(f_{\bm{\theta}})=\prod_{\ell=1}^{L}\|\mathbf{W}_{\ell}(\bm{\theta})\|_{2}$ for Bayesian heads on frozen CIFAR-10 features. The SpectralCirculant1d head stays in a substantially lower Lipschitz regime throughout SVI compared to the Dense head.

Figure 6: Posterior distributions of the global Lipschitz upper bound $\widehat{\operatorname{Lip}}(f_{\bm{\theta}})$ for Dense vs SpectralCirculant1d heads in the deep-kernel setting (frozen CIFAR-10 encoder). Despite similar predictive performance, the spectral head yields much smaller Lipschitz constants, reflecting a smoother dependence on the frozen representation.

Figure 7: Frozen CIFAR-10 features: spectral-circulant SVI heads yield substantially larger certified $\ell_{2}$ radii due to smaller global Lipschitz bounds $\widehat{\operatorname{Lip}}(f_{\bm{\theta}})$ .

Figure 8: Tiny ImageNet test accuracy vs parameter count.

Explore similar work

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
May 4, 2026Lurong PanNeural Network Optimization Feedforward Neural Networks
Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks
Jan 29, 2026Kaustubh Sharma, Srijan Tiwari, Ojasva Nema +1Prior-Data Fitted Networks Neural Network Interpretability
Indian Institute of Technology Roorkee, India
Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Oct 8, 2026Dmitrii Andriianov, Andrey Veprikov, Aleksandr BeznosikovWeight Decay Nuclear Norm Minimization
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab · Innopolis University

May 4, 2026J/K move · Enter open · S save

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

Lurong Pan

Neural Network Optimization Feedforward Neural Networks Neural Network Compression

Open paper AskPDF arXiv

Communication Dynamics Neural Networks (CDNNs) apply the circulant-spectral machinery of the Communication Dynamics framework to neural-network layer design. We introduce CDLinear, a block-circulant linear layer with block size B = 2l + 1 that uses 1/B the parameters of a dense layer with the same input and output dimensions. The construction gives an explicit Fourier-domain diagnostic for optimization: for mean-squared loss, the weight Hessian is diagonalized by the discrete Fourier transform, with eigenvalues determined directly by the Fourier spectrum of the input blocks. Under input pre-whitening, the population Hessian condition number is exactly 1, and the empirical condition number is bounded by 1 + O(sqrt(B/N)) for N samples. We implement CDLinear in pure NumPy with hand-derived backward passes and verify gradients by finite differences. On the 8x8 MNIST digits benchmark, across three random seeds, a CDLinear MLP with B = 4 reaches 97.50% +/- 0.23% test accuracy using 2,380 parameters, compared with 98.15% +/- 0.47% for a dense baseline using 8,970 parameters. This gives a 3.8x parameter reduction at a 0.65% accuracy cost. The CD-MLP's mean Hessian condition number is 1.9e4, about 310x smaller than the dense baseline's 5.9e6. We position CDLinear as a special case of structured matrix neural-network layers, with the main contributions being a closed-form Hessian-spectrum diagnostic, a principled discrete sequence of block multiplicities, and an explicit conditioning analysis. We also release a reference PyTorch implementation integrating CDLinear into a DeepSeek-V3-style mixture-of-experts transformer for future large-scale benchmarks.

Abstract

Explore similar work

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights