Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

Opegbemi Matthias Busoye; Tolulope Matthew Busoye; Eghonghon-aye Eigbe

cs.LGSep 30, 2026

Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

Authors: Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye Eigbe

Organizations: PowerLabs Technologies, Lagos, Nigeria

Abstract

Activation memory, not compute, limits CNN inference on constrained hardware such as microcontrollers. Direct in-place convolution removes the dual-buffer cost, but the memory-optimal formulation of Gural and Murmann assumes valid padding, unit stride, unit dilation, and odd square kernels, and needs a non-sequential traversal costing $2\times$ inference time in transposes. We identify two regimes in which their published closed form does not hold: (1) an under-allocation of exactly $(k-1)C_{in} \bmod (C_{out}-C_{in})$ scalars, active on every convolutional layer of their own deployed network and manifesting as a silent corruption of still-live input; (2) an unbounded overestimate, up to $2{,}432\times$ , once the critical leg leaves the output grid. We correct both and generalize to arbitrary stride, dilation, padding, and rectangular kernels. We then propose Right In-Place (RiP) convolution, a bit-identical operation in which every layer reads its input right-aligned in a shared workspace and writes its output left-aligned from index zero. The debt is piecewise affine in the output pixel index, so evaluating its breakpoints in $O(1)$ yields the minimum safe gap without enumerating the output grid, with row-major access preserved. Across $10{,}000$ random layers RiP produced no corruption, and across 84 convolutional layers from 25 architectures it matches the herringbone workspace exactly on 58 and within 5% on 81, using 24.8% less memory than dual buffering on average. Written into TinyEngine's kernels and deployed to a Raspberry Pi Pico 1 and Pico 2, it cuts peak activation memory across eleven MCUNet models by 12.5 to 33.3% at unchanged cycle counts and bit-identical outputs, raising the number of models that fit the Pico 1's 256 KB SRAM from six to nine.

Figures & tables

Figure 1 : Workspace layout on a Pico 1 for MCUNet-in3 under naive dual buffering, TinyEngine’s scheduler, and RiP. All three are placed by the same MILP memory allocator ( Sec. 4.3 ). On the first two it reaches the same peak as TinyEngine’s first-fit memory allocator [ 20 ] , so the allocator alone does not account for the saving; under in-place convolution first-fit fragments the arena and does not allow the full utilization of the extra memory, so the MILP is what lets RiP’s gap be reclaimed rather than stranded.

Method	Axis	General	Sequential	Bound	Restrictions
Herringbone [ 9 ]	spatial in-place	✓	$\times$	✓	valid, $s{=}1$ , $d{=}1$ , odd $k$ square
Replace [ 9 ]	spatial in-place	✓	✓	$\times$	valid, $s{=}1$ , $d{=}1$ , odd $k$ square
In-place DW [ 20 ]	channel in-place	$\times$	✓	–	depthwise only
MCUNetV2 [ 19 ]	cross-layer patch	✓	–	–	recomputation overhead
RAMAN [ 18 ]	hardware overlay	✓	–	$\times$	requires custom accelerator
CMSIS-NN Replace [ 25 ]	inter-layer arena	✓	✓	$\times$	whole-tensor granularity

Table 1: Memory-reduction axes for CNN inference on tiny devices. “General” = applies to standard channel-mixing convolutions; “Sequential” = preserves row-major output order without reordering; “Bound” = derives a closed-form minimum workspace for the layer.

Panel of Figure 2 — Figure 2 : Convolution traversal order.

$C_{in}$	$C_{out}$	$k$	$H{=}W$	$x^{*}$	G&M	Truth	Err.
Limitation 1: G&M’s deployed MNIST network, conv_0–conv_2
1	5	3	14	0.5	531	533	$-2$
5	8	3	12	3.3	148	149	$-1$
8	11	3	10	5.3	45	46	$-1$
Limitation 2: critical leg exceeds the output grid
3	4	5	8	12	79	7	$+72$

Table 2 : The two limitations in the as-published herringbone formula ( Eq. 1 ). Truth is the minimum corruption-free gap, found by binary search against the simulator; Err. is G&M minus Truth, so negative entries are under-allocations. Corollary 1 reproduces Truth on every row.

H{=}W

is the input spatial size.

Model	Buffer (B)	Base	RiP	Ratio	Base	RiP	Ratio	Base	RiP	Ratio	Acc.	Bit-exact
		Peak activation memory (B)			Pico 2 cycles (M)			Pico 1 time (ms)
MCUNet-in0	3,984	46,080	36,864	${4}{/5}$	39.5	39.7	$1.003$	1,772.4	1,772.0	$1.000$	$40.4\%$	✓
MCUNet-in1	5,864	92,160	73,728	${4}/{5}$	70.0	70.3	$1.004$	3,601.1	3,607.8	$1.002$	$49.9\%$	✓
MCUNet-in2	14,976	204,800	179,200	${7}/{8}$	307.0	307.2	$1.001$	15,978.1	15,999.5	$1.001$	$60.3\%$	✓
MCUNet-in3	18,536	247,808	185,856	${3}/{4}$	380.5	380.8	$1.001$	–	19,388.2	–	$61.8\%$	✓
MCUNet-in4	16,520	409,600	307,200	${3}/{4}$	708.2	708.7	$1.001$	–	–	–	$68.0\%$	✓

Table 3 : Eleven models from the MCUNet suite, baseline versus RiP. Buffer is the scratch and im2col allocation, which sits outside the arena and is unchanged by RiP. Peak activation memory is the arena alone, excluding that buffer, the SDK and TinyUSB .data / .bss , and the stack; it is a property of the allocation, so it is reported once for both boards. Cost is cycles count on the Pico 2 and wall-clock time on the Pico 1, whose RP2040 exposes no cycle counter. A dash marks a model the board could not run, since the arena is not the only claim on memory. Acc. is int8 top-1 as reported by [ 20 ] , identical under both strategies;

\dagger

marks Visual Wake Words rather than ImageNet, and

\ddagger

a single-class detector, for which no accuracy is reported. Bit-exact reports agreement of on-device output checksums against baseline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Convolution setup ( Sec. 4.1 )
$\mathbf{X},\mathbf{Y}$	Input, output activation tensors
$H_{in},W_{in},C_{in}$	Input height, width, channel count
$H_{out},W_{out},C_{out}$	Output height, width, channel count
$H_{p},W_{p}$	Padded input height, width
$N_{out}$	Output tensor size, $H_{out}W_{out}C_{out}$
$k_{h},k_{w}$	Kernel height, width

Appendix

Table 4 : Symbols used throughout the paper, grouped by the section that introduces them.

D

is overloaded by arity:

D(r,c)

is the row-major RiP debt of Eq. 7 ,

D(x)

the herringbone leg debt of Eq. 11 .

Panel of Figure 3 — Figure 3 : Herringbone traversal order on uneven output grids.

Tier	Generator	Share
1	plain convolution, $k\in[1,7]$	$11\%$
1	same padding, $p=\lfloor k/2\rfloor$	$8\%$
1	strided downsampling	$6\%$
1	inverted-bottleneck expansion	$5\%$
1	projection, $C_{out}\leq C_{in}$	$5\%$
2	collapsed minor axis, $n\in\{1,2,3\}$	$9\%$

Appendix

Table 5 : The twenty generators, with their share of the sweep. Tier 1 is the geometry shipping models are built from, tier 2, the small-grid and degenerate regimes where every defect found so far has lived, and tier 3, the correctness corners that generality requires but real models rarely reach.

Family	$n$	Gen.	Exact	${\leq}5\%$	Max	Save	Dense	Timed	Cost
ResNet-50	11	4	10	11	$1.006$	$25.5\%$	$2.3\%$	6	$1.000$
ResNet-56 (pruned)	5	5	4	5	$1.011$	$26.9\%$	$11.4\%$	5	$1.001$
VGG-16	4	4	0	4	$1.033$	$25.0\%$	$0.9\%$	–	–
VGG-16 (pruned)	3	3	0	1	$1.086$	$17.9\%$	$4.2\%$	3	$1.000$
MobileNetV1	6	1	5	6	$1.005$	$32.1\%$	$0.0\%$	3	$1.000$
MobileNetV2	5	1	4	5	$1.005$	$16.8\%$	$0.0\%$	2	$1.000$

Appendix

Table 6 : RiP against the generalized herringbone bound on 84 convolutional layers drawn from 25 architectures.

n

is layers sampled; Gen. counts those whose parameters fall outside the restrictions of [ 9 ] and so are reachable only through Theorem 11 . Exact and

{\leq}5\%

count layers where RiP’s workspace equals, or is within 5% of, herringbone’s. Max is the worst RiP-to-herringbone workspace ratio in the family. Save is the mean workspace reduction against dual buffering, and Dense the mean further reduction from not materializing padding ( Appendix F ). Timed counts the layers executed on the Pico 2 and Cost is their median runtime ratio against the dual-buffer kernel, right-shift included; a dash marks a family with no timed layer. Every layer passes the corruption simulator under both layouts, so no correctness column is shown.

Model	Ops	Res.	Vars	Constr.	Dual buf.	RiP	Ratio	Dual buf.	RiP	Ratio	In-place (%)	Shift (#)
			MILP size		Compile (s)			Peak activation (KB)			Tensor placement
MobileNet	29	0	1,361	1,969	0.08	0.14	1.69	1,176.0	784.0	2/3	60.0	–
MobileNetV2	64	10	6,364	8,813	0.26	0.52	2.01	1,470.0	1,176.0	4/5	49.2	–
MobileNetV3-S	109	15	18,201	24,834	1.08	1.49	1.37	392.0	220.5	9/16	64.5	–
MobileNetV3-L	122	18	22,751	30,967	1.27	1.42	1.12	980.0	784.0	4/5	82.1	–
VGG-16	21	0	729	1,077	0.11	0.16	1.49	6,272.0	3,150.1	0.502	54.5	1

Appendix

Table 7 : Graph-level allocation across 33 keras.applications models under the dense layout with the receptive field gathered first. Ops is graph nodes after folding and Res. the branch connections that keep two tensors alive at once. Vars and Constr. size the MILP; Compile is the full offline step (parse, model build, solve) under each formulation, median of 10 runs. Dual buffer is the same program restricted to disjoint placement, so it is the optimal packing of never-aliased tensors rather than a greedy baseline. In-place is the share of tensors the solver places in-place and Shift the number it separates by a physical right shift; a dash marks a model with no move scheduled. The memory ratio is an exact fraction on 29 of the 33, and the four exceptions are exactly the four models with a non-empty Shift column.

Explore similar work

May 10, 2026cs.DC

Split CNN Inference on Networked Microcontrollers

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Junyu Lu, Shashwath Suresh, Hao Liu +2

Delft University of Technology, Delft, The Netherlands

Neural Network Inference Cortex Convolutional Neural Networks

Ask PDF

Jul 2, 2026cs.LG

Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency

Newer lightweight convolutional neural networks are often presented as improving predictive performance and deployment efficiency, but such claims require controlled evaluation. This study compares nine lightweight CNN model packages across CIFAR-10, CIFAR-100, and Tiny ImageNet under a shared downstream protocol. We report top-1 accuracy, macro F1, top-5 accuracy, parameter count, FP32 storage, GMACs, batch-size-1 latency on an NVIDIA L4 and AMD Ryzen 5 5500U CPU, peak PyTorch CUDA allocated tensor memory, and point estimate Pareto frontiers. EfficientNetV2-S achieves the highest observed top-1 accuracy on CIFAR-10 and CIFAR-100 at 97.57% and 86.98%, while RepViT-M1.0 leads Tiny ImageNet at 79.87%. EfficientNet-B0 remains within 0.22, 0.85, and 1.79 percentage points of the best result on the three datasets while using approximately 79% fewer parameters and 86% fewer GMACs than EfficientNetV2-S. It also appears on every evaluated accuracy and resource Pareto frontier, making it the most consistently competitive intermediate-budget option. MobileNetV3-Small has the lowest GMAC count, is the fastest model under both CPU thread settings, and records higher observed accuracy than MobileNetV4-Conv-S on all three datasets. Under random initialization, it leads MobileNetV4-Conv-S by 2.55, 1.76, and 0.99 points, with paired test-set intervals excluding zero for the fixed trained models. EfficientNet-B0 remains 3.29, 10.10, and 17.54 points below its pretrained counterpart after 100 epochs of scratch training, despite requiring about five times the recorded training time. SqueezeNet1.1 has the fewest parameters and lowest peak CUDA allocation, but substantially weaker accuracy. Latency rankings differ sharply between the L4 and CPU environments, showing that GMACs alone do not predict measured inference performance. Overall, newer designs provide selective rather than universal gains

Tasnim Shahriar

Independent Researcher

Mobilenetv2 Neural Network Inference Imagenet Multi-Model Study

Ask PDF

Sep 16, 2026cs.LG

Radio-Frequency Convolutional Neural Networks

Running artificial intelligence (AI) models directly on edge devices such as smartphones, wearables, and drones offers low latency, pervasive scalability, and data privacy, but these devices rarely carry the computing capability that modern neural networks demand. Edge accelerators have been developed in response, yet each adds computing hardware to devices already constrained in size, weight, power, and cost (SWaP-C). An alternative lies in what these devices already carry: the frequency mixer in every wireless radio multiplies signals in time, natively performing convolution in the frequency domain. Here we introduce radio-frequency convolutional neural networks (RF-CNNs), which repurpose existing communication hardware for CNN inference. Multi-channel convolutions are mapped onto frequency tones for a passive mixer to execute in a single pass. We experimentally demonstrate that RF-CNN runs deep CNNs up to 26.4 million parameters and nine layers from classification of wireless signals and images to controllable image generation, close to full-precision performance. Because the weights arrive over the air and the analog hardware is shared with communication, the edge device spends energy only on data preparation and readout-down to 0.72 femtojoules per multiply-accumulate, two orders of magnitude less than it would cost on an added digital processor. These results suggest that deployed wireless infrastructure can bring efficient, state-of-the-art AI inference to the billions of devices it already connects.

Zhihui Gao, Shi-Yuan Ma, Yiran Chen +2

Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA. · Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.

Edge Devices Neural Network Inference Open Radio Access Networks Full-Precision Performance

Ask PDF