From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy

Petro Shulzhenko; Gabriele Spadaro; Enzo Tartaglione

cs.LGSep 4, 2026

From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy

Authors: Petro Shulzhenko, Gabriele Spadaro, Enzo Tartaglione

Organizations: LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · University of Turin, Italy

Abstract

Although Deep Neural Networks have become foundational in many areas of Machine Learning, high computational demands limit their application in resource-constrained environments. To address this issue, depth compression methods have been proposed to identify and linearize redundant activation functions, thereby allowing for the merging of layers without intermediate non-linearities. However, these methods face two key challenges: they cannot be directly applied to convolutions with padding due to the absence of an analytical solution for merging these layers, and they typically increase the kernel size of merged layers, thus limiting speed-up gains. To overcome these limitations, we propose an efficient strategy that enables merging of layers without an existing analytical solution, and also without increasing kernel size. We validate our approach across multiple architectures and datasets, and measure inference speed-up gains on real embedded platforms. We publicly released the code at https://github.com/ShulzhenkoPetr/deep-to-shallow.

Figures & tables

Figure 1: Accuracy vs. speed-up for ResNet18 on ImageNet-1k. Unlike the Reference model, the proposed Shrunk model converts additional linearized activations into actual speed-up while preserving accuracy.

Figure 2: Illustration of the proposed method. Layers connected through linearized activations are merged and replaced by a single trainable layer. Only these layers are optimized during fine-tuning, while all others are kept frozen.

Method	Kernel size	Loss	Shrunk acc.
Analytical solution	5	–	91.61
Ours	5	$\mathcal{L}_{A}$ ( 3 )	91.07
	3	$\mathcal{L}_{A}$ ( 3 )	90.61
	3	$\mathcal{L}_{Comb}$ ( 4 )	91.28

Table 1: CIFAR-10 accuracy of ResNet18 Shrunk models with analytical and loss-based merging under different kernel sizes.

Model	Ref.	#	Shrunk	$(T_{\text{ref}}-T_{\text{shrunk}})\uparrow$ (ms)
Model	acc.	Merged	acc.	Raspi5	Jetson	2080Ti
ResNet18	63.10	3	64.20	+17.0	+1.8	+0.5
MobileNetv2	70.44	3	70.47	+2.0	+1.9	+0.6
ResNet50	74.93	2	75.08	+30.0	+2.3	+0.8
Swin-T	77.54	2	77.51	+28.0	+1.2	+0.3

Table 2: ImageNet accuracy and inference-time reduction of Reference and Shrunk models across architectures and devices.

Method	# Merged	Ref. acc.	Shrunk acc.
Smallest weights	2	92.20	92.27
Smallest gradients	2	92.35	92.40
TLC	2	92.63	92.67
EASIER	2	92.85	92.84

Table 3: CIFAR-10 accuracy of Reference and Shrunk ResNet18 models with different activation linearization methods.

Model	Kernel	Ref.	Shrunk	$(T_{\text{ref}}-T_{\text{shrunk}})\uparrow$ (ms)
Model	Kernel	acc.	acc.	Raspi5	Jetson	2080Ti
ResNet18	5	92.68	92.69	-3.2	-1.5	-0.1
	3	92.68	92.32	+10.4	+0.4	+0.9
	DS3	92.68	91.79	+16.3	+2.5	+0.9

Table 4: CIFAR-10 test accuracy of ResNet18 Shrunk models with 7 linearized activations and 4 merged blocks, evaluated across different kernel sizes used in the merged convolutional layers.

Model (#)	Kernel	Raspi5	J. Nano	J. Orin
R18 (3)	$3{\times}3$	$\times 1.12$	$\times 1.12$	$\times 1.14$
R18 (3)	$5{\times}5$	$\times 1.01$	$\times 0.79$	$\times 1.14$
R34 (6)	$3{\times}3$	$\times 1.14$	$\times 1.15$	$\times 1.17$
R34 (6)	$5{\times}5$	$\times 0.96$	$\times 0.74$	$\times 1.07$

Table 5: ImageNet inference speed-up (

T_{\text{ref}}/T_{\text{shrunk}}

, batch 1) of ResNet Shrunk models, for the merged-layer kernel size

3{\times}3

(ours) and the

5{\times}5

kernel an analytical merge requires. The merged architecture is identical in both rows, isolating the effect of kernel growth.

Figure 3: Test accuracy and convergence time of ResNet18 Shrunk models under different training strategies.

Model	Reference	$\mathcal{L}_{\text{A}}$	$\mathcal{L}_{\text{D}}$	$\mathcal{L}_{\text{CE}}$	Shrunk
Model	acc.	$\mathcal{L}_{\text{A}}$	$\mathcal{L}_{\text{D}}$	$\mathcal{L}_{\text{CE}}$	acc.
ResNet18	92.68	–	–	✓	90.26
		–	✓	–	92.27
		✓	–	–	10.00
		✓	–	✓	90.39
		✓	✓	–	92.32

Table 6: Ablation of loss components for ResNet18 Shrunk models on CIFAR-10 and Tiny ImageNet.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 218 papers

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual Learning Neural Network Optimization
ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455
Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers
Sep 4, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeNeural Network Compression On-Device Inference
PowerLabs Technologies, Lagos, Nigeria
HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks
Jun 29, 2026Lukas Meiner, Jens Mehnert, Alexandru Paul ConduracheEfficient Neural Network Inference Convolutional Neural Networks
Robert Bosch GmbH, Leonberg, Germany. · Universität zu Lübeck, Lübeck, Germany.

Feb 9, 2026J/K move · Enter open · S save

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Yilang Zhang, Bingcong Li, Niao He, Georgios B. Giannakis

ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455

Residual Learning Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (

<1\%

), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.

Abstract

Neural Network Compression

Explore similar work

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers

HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks