Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs

Pedram Bakhtiarifard; Tong Chen; Jonathan Wenshøj; Erik B Dam; Raghavendra Selvan

cs.LGFeb 16, 2026

Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs

Authors: Pedram Bakhtiarifard, Tong Chen, Jonathan Wenshøj, Erik B Dam, Raghavendra Selvan

Organizations: Department of Computer Science, University of Copenhagen, Denmark

Abstract

Large-scale deep learning models are well-suited for compression. Across a variety of tasks, methods like pruning, quantization, and knowledge distillation have been used to achieve massive reductions in model parameters with only marginal performance drops. This raises the central question: Why are deep neural networks suited for compression? In this work, we take up the perspective of algorithmic complexity to explain this behavior. We hypothesize that the parameters of trained models have more structure and, hence, exhibit lower algorithmic complexity compared to the weights at (random) initialization. Furthermore, model compression methods harness this reduced algorithmic complexity to compress models. Although an unconstrained parameterization of model weights, $\mathbf{w} \in \mathbb{R}^n$ , can represent arbitrary weight assignments, the solutions found during training exhibit repeatability and structure, making them simpler to implement than a trivial program. To this end, we formalize the Kolmogorov complexity of $\mathbf{w}$ by $\mathcal{K}(\mathbf{w})$ . We introduce a constrained parameterization $\widehat{\mathbf{w}}$ that partitions parameters into blocks of size $s$ and restricts each block to be selected from a set of $k$ reusable motifs, specified by a reuse pattern (or mosaic). The resulting method, $\mathit{Mosaic\text{-}of\text{-}Motifs}$ (MoMos), provides a theoretically justified parameterization that biases optimization toward algorithmically simpler solutions. Empirical evidence from multiple experiments shows that MoMos consistently lowers the algorithmic complexity of neural networks during training while preserving the performance of unconstrained models. These results suggest that parameter compressibility is not only observed after training, but can be induced from the optimization domain.

Figures & tables

Panel of Figure 1 — Figure 1: An unconstrained (left) $16\times 16$ parameterization with randomly selected motifs highlighted. MoMos parameterization (right) enforcing $2\times 2$ motif reuse.

Figure 2: Algorithmic complexity reduction for randomly sampled pretrained timm models with up to 100M parameters, measured as the ratio of pretrained to initialized complexity using Eq. ( 2 ). The 200 models are ordered by increasing complexity reduction.

Figure 3: The algorithmic complexity makeup attributed to motifs vs. mosaic arrangement. Bit allocation (bottom) share of motifs contributes most of the still lower (top) relative complexity.

Figure 4: The effect of block size granularity on domain geometry. Larger (top) blocks subsume the flexibility of local patterns impacting the richness of the (bottom) optimization subspaces of the domain; algorithmic simplification is exchanged for function multiplicity.

Figure 5: Trade-off between algorithmic simplification and richness of the optimization domain for Tiny-ViT ( Wu et al., 2022 ) using MoMos. For a fixed block size $s=4$ (left) , capacity $c=k/m$ is the fraction of blocks retained as motifs; increasing $c$ enriches the optimization domain while reducing the amount of algorithmic simplification. Across block sizes $s$ (right) , darker colors indicate a richer optimization domain, and white contours mark fixed areas of simplification.

Figure 6: Ablation over capacity and block size for Tiny-ViT on (top) test accuracy and (bottom) RAS.

Epoch	Top $1\%$	Top $10\%$	Top $25\%$
1	$3.09$	$21.79$	$44.30$
200	$5.39$	$34.59$	$64.19$

Table 1: Mosaic share of most used motif for Tiny-ViT

(s=4,c=0.1)

Figure 7: Tiny-ViT distortion and validation loss at convergence.

Figure 8: Loss landscape across architectures. For each model, we take the converged baseline parameters and two MoMos solutions with $s=2$ and $s=4$ at fixed capacity $c=0.1$ , and evaluate validation cross-entropy on the plane spanned by these three parameter vectors. The coordinates $(\alpha,\beta)=(0,0)$ , $(1,0)$ , and $(0,1)$ correspond to the baseline, $s=2$ , and $s=4$ solutions, respectively. Color shows CE loss normalized within each panel, and contours show fixed areas of loss.

Figure 9: Motif assignment concentration is shown for MoMos with

s=2,c=0.1

. From epoch 1 to 200, assignment mass shifts toward the most-used motifs; the bottom 50% receive only a small fraction of assignments at convergence.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Cap. (%)	Config.	MLP			ResNet20			Tiny-ViT
			Acc. (%) $\uparrow$	RAS $\uparrow$	$\mathbf{r}_{\mathbf{BDM}}$ $\downarrow$	Acc. (%) $\uparrow$	RAS $\uparrow$	$\mathbf{r}_{\mathbf{BDM}}$ $\downarrow$	Acc. (%) $\uparrow$	RAS $\uparrow$	$\mathbf{r}_{\mathbf{BDM}}$ $\downarrow$
Baseline	100	$q=32$	$56.40\pm 1.70$	1.00	0.96	$92.20\pm 0.50$	1.00	0.90	$86.60\pm 0.20$	1.00	1.00
QAT	–	$q=16$	$55.80\pm 3.80$	2.00	0.97	$92.40\pm 0.30$	2.00	0.90	$87.00\pm 0.20$	2.00	1.00
QAT	–	$q=8$	$57.30\pm 1.90$	4.00	0.96	$92.30\pm 0.50$	4.00	0.90	$87.10\pm 0.20$	4.00	1.00
QAT	–	$q=4$	$55.00\pm 3.20$	8.00	0.86	$91.80\pm 0.30$	8.00	0.74	$86.90\pm 0.30$	8.00	0.97
QAT	–	$q=2$	$31.50\pm 1.60$	16.00	0.07	$13.40\pm 2.50$	16.00	0.02	$74.20\pm 0.30$	16.00	0.54

Appendix

Table 2: CIFAR-10 results for MoMos and several compression baselines. We report test accuracy, relative algorithmic simplification (RAS), and the BDM ratio computed on the binarized weights as in Eq. ( 2 ). MoMos configurations vary block size

s

and capacity

c

, while baseline compression methods are evaluated using the settings described in Appx. B .

Method	Cap. (%)	Config.	MLP		ResNet20		Tiny-ViT
			Time $\Delta$	Mem. $\Delta$	Time $\Delta$	Mem. $\Delta$	Time $\Delta$	Mem. $\Delta$
Baseline	100	$q=32$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$
MoMos	10	$s=2$	$5.30$	$203.02$	$-3.10$	$0.00$	$3.29$	$-0.00$
MoMos	10	$s=4$	$3.00$	$105.92$	$-2.99$	$0.00$	$1.85$	$-0.00$
VQ	–	$k=16$	$3.27$	$294.49$	$-2.45$	$0.00$	$-0.30$	$0.00$
PQ	–	$k=8,d=2$	$12.94$	$0.10$	$-2.96$	$0.00$	$7.78$	$0.00$

Appendix

Table 3: Training-time and memory overhead between MoMos and baselines compression methods. Overheads are reported as percentages relative to the FP32baseline for the same architecture, measured in separate 20-epoch profiling runs with the first 5 epochs treated as warm-up epochs. Methods with projection steps use the Flash projection approach from Yang et al. (2026) for consistency.

Method	Cap. (%)	Config.	SmolLM2 (360M params.)		Swin-B (86M params.)
			PPL $\downarrow$	RAS $\uparrow$	Acc. (%) $\uparrow$	RAS $\uparrow$
Baseline	100	$q=32$	$9.55\pm 0.00$	1.00	$90.35\pm 0.17$	1.00
QAT	–	$q=16$	$9.55\pm 0.00$	2.00	$90.40\pm 0.20$	2.00
QAT	–	$q=8$	$10.89\pm 0.00$	4.00	$90.30\pm 0.10$	4.00
MoMos	5	$s=2$	$9.86\pm 0.09$	2.35	$90.38\pm 0.07$	2.54
MoMos	10	$s=2$	$9.74\pm 0.06$	2.04	$90.61\pm 0.06$	2.18

Appendix

Table 4: MoMos fine-tuning results on SmolLM2 and Swin-B. SmolLM2 is fine-tuned for 3 epochs and evaluated with perplexity, while Swin-B is fine-tuned for 30 epochs and evaluated with test accuracy. QAT baselines use weight-only quantization-aware training at the indicated bitwidths.

Figure 10: Ablation over capacity and block size for (top) validation accuracy and (bottom) RAS. The dashed horizontal line is the FP32 baseline. Smaller motif blocks achieve baseline accuracy at lower capacity, whereas larger blocks require higher capacity to reach comparable performance.

Figure 11: Motif assignment concentration is shown for MoMos with

s

equal 2 and 4 for

c=0.1

. Motifs are ranked by usage, grouped by percentile, and the bars indicate the fraction of block assignments for each group. The assignment mass shifts toward the most used motifs, while the bottom 50% receive only a small fraction of assignments at convergence.

Figure 12: Distortion and validation loss decrease together. Each arrow points from and towards the first epoch and last epoch (at convergence) for a specific MoMos setting. Most settings result in a reduction in both the projection distortion and a lower validation loss.

Figure 13: Distortion and validation loss during training for the MLP across MoMos settings. Each panel fixes the block size

s

and capacity

c

Figure 14: Distortion and validation loss during training for ResNet20 across MoMos settings. Each panel fixes the block size

s

and capacity

c

Figure 15: Distortion and validation loss during training for Tiny-ViT across MoMos settings. Each panel fixes the block size

s

and capacity

c

Kolmogorov Complexity

See all 12 papers

May 15, 2026

Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis

Training large-scale deep neural networks (DNNs) is resource-intensive, making model compression a practical necessity. The widely accepted ''learning as compression'' hypothesis posits that training induces structure in network weights, which enables compression. Measuring this structure through Kolmogorov-Chaitin-Solomonoff (KCS) complexity is appealing, but existing estimators based on the Coding Theorem Method (CTM) and the Block Decomposition Method (BDM) are limited to small binary objects and do not scale to modern DNNs. We introduce the Quantized Block Decomposition method (QuBD), which extends algorithmic complexity estimation to any

k

-ary object. QuBD first quantizes the network weights to a finite alphabet, then estimates the KCS complexity by aggregating per bit-plane CTM estimates. We show theoretically that QuBD yields a strictly tighter estimation gap with respect to true KCS complexity than binarization-based methods. Using QuBD, we study how the algorithmic complexity of neural network weights evolves during training, showing that it decreases as models learn, scales with data budget, increases during overfitting, follows the delayed generalization observed during grokking, and correlates with generalization performance. We further show that algorithmic information resides predominantly in the most significant bit-planes, which can serve as a practical diagnostic for determining appropriate post-training quantization levels. This work offers novel insights into learning mechanisms in DNNs by providing the first scalable, tractable estimates of KCS complexity for large, non-binary objects such as DNN weights.

Pedram Bakhtiarifard, Sophia N. Wilson, Mahmoud Afifi +2

Department of Computer Science, University of Copenhagen, Denmark · The American University in Cairo, Egypt

Neural Network Generalization Neural Network Training Dynamics

Ask PDF

Oct 8, 2026

Dynamics as Code: On Model Compression via Dynamic System

The escalating size of pretrained neural networks has rendered model compression a prerequisite for deployment under stringent memory and compute constraints. With the irrational winding as an example, earlier work introduced a dynamic system (DS) paradigm that reconceptualizes compression as compact weight representation: high-dimensional parameters are encoded by the index of a trajectory produced by a dynamic system, from which the vector is recovered during decompression. This mechanism is fundamentally distinct from pruning, quantization, knowledge distillation, and low-rank decomposition. Along this direction, we prove that under a Diophantine condition, a finite trajectory of

M = O(ε^{-(d+ν)})

states in the irrational winding constitutes an

ε

-net over the

d

-dimensional weight space, thereby linking state resolution, decompression error, and compression ratio in a predictable manner. Furthermore, we propose a generalized DS-based model compression framework by unifying four DS families---space-filling curves (Hilbert, Peano, Morton/Z-order, Snake), chaotic systems (Lorenz), congruential and pseudo-random generators (LCG, PCG), and low-discrepancy sequences (Halton). Also, we introduce the KD-tree and coordinate-template acceleration to scale to large models as well as outlier identification to control the error. Experiments on ResNet-18 and Qwen2.5-1.5B/Qwen1.5-7B validate that DS-based compression achieves competitive compression ratios without post-hoc retraining, with controllable decompression error and flexible state-space design, establishing it as a principled and practical compression approach.

Fan Gao, Wei Su, Juntong Fan +4

School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu Province, China · Department of Data Science, City University of Hong Kong, Hong Kong, China SAR · Department of Mathematics, City University of Hong Kong, Hong Kong, China SAR +1

Model Compression Neural Network Compression LLM Compression

Ask PDF

Mar 31, 2026

Big2Small: A Unifying Neural Network Framework for Model Compression

With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed Big2Small, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. Big2Small trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that Big2Small achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.

Jing-Xiao Liao, Haoran Wang, Tao Li +4

Frontier of Artificial Networks (FAN) Lab, Department of Data Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · Hong Kong Applied Science and Technology Research Institute (ASTRI) · Sichuan Engineering Research Center for Big Data Visual Analytics, Sichuan University of Science and Engineering, Zigong, China +3

Model Compression Neural Network Compression

Ask PDF

Abstract

Kolmogorov Complexity

Explore similar work

Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis

Dynamics as Code: On Model Compression via Dynamic System

Big2Small: A Unifying Neural Network Framework for Model Compression