Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning

Zhenglin Huang; Qifa Yan; Bin Dai; Xiaohu Tang

cs.LGAug 21, 2026

Lightweight Adaptive ReduNet via Hyperspherical Manifold Learning

Authors: Zhenglin Huang, Qifa Yan, Bin Dai, Xiaohu Tang

Organizations: School of Information Science and Technology, Southwest Jiaotong University, Chengdu 610031, China · Information Coding and Transmission Key Laboratory of Sichuan Province, CSNMT Int. Coop. Res. Centre (MoST), Southwest Jiaotong University, Chengdu 611756, China

Abstract

In recent years, a white-box neural network called ReduNet has been proposed, which employs the maximal coding rate reduction (MCR $^2$ ) principle to transform raw data into low-dimensional discriminative features via a forward layer-wise construction process. Unlike traditional deep networks that rely on backpropagation, ReduNet explicitly derives the parameters of each layer from the features of its preceding layer, offering a mathematically interpretable paradigm. However, this layer-wise construction often requires a large number of layers for the MCR $^2$ objective to reach a stable value, which increases the parameter storage of the unfolded module. To address this issue, we propose LA-ReduNet, a lightweight adaptive architecture that refines the layer-wise update rule and enables discriminative feature representations to be obtained with substantially fewer unfolded layers. Specifically, LA-ReduNet employs hyperspherical manifold learning and adaptive step sizes, thereby reducing by an order of magnitude the number of layers required for the MCR $^2$ objective to reach a stable value. Simulation results demonstrate that, while maintaining comparable classification accuracy, LA-ReduNet requires significantly fewer layers for the MCR $^2$ objective to reach a stable value. Remarkably, under the considered experimental settings, LA-ReduNet requires only approximately $1/29$ of the parameter storage of the unfolded ReduNet module for the MCR $^2$ objective to reach a stable value.

Figures & tables

Panel of Figure 1 — Figure 1: Single-Layer Structure of ReduNet.

Panel of Figure 2 — Figure 2: Orthogonal decomposition of the iterative update on the sphere $\mathbb{S}_{n-1}$ .

Panel of Figure 3 — Figure 3: Example of gradient-based update on the unit sphere $\mathbb{S}^{1}$ : (a) when $||\bm{g}_{\rm{T}}||_{2}\leq\tau$ , the update magnitude in the direction of $\bm{g}_{\rm{T}}$ becomes extremely small, thus we terminate the update of $\bm{z}^{(\ell)}$ ; (b) when $||\bm{g}_{\rm{T}}||_{2}>\tau$ , the update of feature $\bm{z}^{(\ell)}$ is accelerated by $t$ , where the adaptive step-size parameter lies in $[t_{0},t_{0}(1+\beta)]$ .

Panel of Figure 4 — Figure 4: Feature Extraction Process of LA-ReduNet.

Figure 5: Structure of the dimension-reduction module.

	CIFAR-10	CIFAR-100	CINIC-10
Training set	50000	50000	90000
Test set	10000	10000	90000
Number of classes	10	100	10
Data dimension	32 $\times$ 32 $\times$ 3	32 $\times$ 32 $\times$ 3	32 $\times$ 32 $\times$ 3

Table 1: Properties of the evaluated datasets.

Panel of Figure 6 — Figure 6: Trends of classification accuracy with the number of iterative layers $L$ for ReduNet, AR-ReduNet, and LA-ReduNet: (a) CIFAR-10; (b) CIFAR-100. Notice that the accuracy at $L=0$ corresponds to the result obtained by convolutional module.

Panel of Figure 7 — Figure 7: Curves of MCR 2 objective values during training: (a) CIFAR-10; (b) CIFAR-100.

Panel of Figure 8 — Figure 8: t-SNE visualization of learned features on the CIFAR-10 dataset.

Panel of Figure 9 — Figure 9: t-SNE visualization of learned features on the CIFAR-100 dataset.

Figure 10: The trends of classification accuracy versus the number of iterative layers L for ReduNet, AR-ReduNet, and LA-ReduNet on the CINIC-10 dataset.

Figure 11: Curves of MCR 2 objective values during training on CINIC-10.

Panel of Figure 12 — Figure 12: t-SNE visualization of learned features on the CINIC-10 dataset.

Panel of Figure 13 — Figure 13: Effect of different components on the convergence of the MCR 2 objective.

Variant	Update direction	NT	Step-size parameter
Riemannian (Fixed)	$\bm{g}_{\mathrm{T}}$	No	$t=t_{0}$
Riemannian + NT (Fixed)	$\bm{g}_{\mathrm{T}}/\\|\bm{g}_{\mathrm{T}}\\|_{2}$	Yes	$t=t_{0}$
LA-ReduNet (Adaptive)	$\bm{g}_{\mathrm{T}}/\\|\bm{g}_{\mathrm{T}}\\|_{2}$	Yes	Eq. ( 21 )

Table 2: Configurations of the variants used in the component-wise ablation study.

Panel of Figure 14 — Figure 14: Effect of parameter $t_{0}$ on the convergence of the MCR 2 objective of LA-ReduNet.

Panel of Figure 15 — Figure 15: Effect of parameter $\beta$ on the convergence of the MCR 2 objective of LA-ReduNet.

Weight coefficient $\mu$	CIFAR-10	CIFAR-100	CINIC-10
$10^{-1}$	44.71%	9.28%	30.13%
$10^{-2}$	84.66%	55.97%	72.86%
$\mathbf{10^{-3}}$	87.58%	62.78%	76.79%
$10^{-4}$	86.48%	61.84%	74.49%
$0$	86.18%	61.67%	75.20%

Table 3: Ablation study on the weight coefficient

\mu

in terms of classification accuracy on CIFAR-10, CIFAR-100, and CINIC-10.

Panel of Figure 16 — Figure 16: Ablation study on the weight coefficient $\mu$ , illustrating how the final CE loss and MCR 2 objective value vary with $\mu$ .

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 218 papers

Replacement Learning: Training Neural Networks with Fewer Parameters
May 19, 2026Yuming Zhang, Peizhe Wang, Tianyang Han +5Deep Learning Optimization Neural Network Compression
The University of Hong Kong, Hong Kong, China · Southeast University, Nanjing, China
Simplifying Neural Networks During Training
Jul 30, 2026Lorenzo Sciandra, Samuele Fonio, Roberto EspositoNeural Network Training Dynamics Neural Network Compression
Department of Computer Science, University of Turin, Turin, Italy
Random Cloud: Finding Minimal Neural Architectures Without Training
Apr 29, 2026Javier Gil BlázquezNeural Architecture Search Feedforward Neural Networks

May 19, 2026J/K move · Enter open · S save

Replacement Learning: Training Neural Networks with Fewer Parameters

Yuming Zhang, Peizhe Wang, Tianyang Han, Hengyu Shi+4

The University of Hong Kong, Hong Kong, China · Southeast University, Nanjing, China

Deep Learning Optimization Neural Network Compression

Open paper AskPDF arXiv

End-to-end training with full-depth backpropagation remains the dominant paradigm for optimizing deep neural networks, but its efficiency deteriorates as models grow deeper. Since every block must be executed and differentiated under a single global objective, full-depth BP introduces substantial parameter redundancy, activation-memory cost, and training latency, especially when neighboring layers exhibit highly correlated learning patterns. Directly skipping or removing layers can reduce cost, but often weakens representation capacity or requires architecture-specific reuse designs. In this paper, we propose Replacement Learning (RepL), a training-time paradigm that reduces full-depth redundancy by replacing selected blocks rather than simply discarding them. For each removed block, RepL inserts a lightweight computing layer that synthesizes a surrogate operator from the parameters of its adjacent preceding and succeeding blocks through a learnable transformation, and applies the synthesized operator to the preceding activation. In this way, RepL preserves local contextual continuity while avoiding unnecessary full-layer computation. We instantiate RepL for CNNs and ViTs with tailored parameter-fusion blocks that handle convolutional channels, feature resolutions, and transformer submodules. Extensive experiments on CIFAR-10, SVHN, STL-10, ImageNet, COCO, and CityScapes show that RepL reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training across classification, detection, and segmentation. Additional results on WikiText-2, transfer learning, inference throughput, checkpointing, stochastic depth, and INT8 quantization further demonstrate its generality and compatibility.

Abstract

Neural Network Compression

Explore similar work

Replacement Learning: Training Neural Networks with Fewer Parameters

Simplifying Neural Networks During Training

Random Cloud: Finding Minimal Neural Architectures Without Training