Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

Opegbemi Matthias Busoye; Tolulope Matthew Busoye; Eghonghon-aye Eigbe

cs.LGSep 30, 2026

Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

Authors: Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye Eigbe

Organizations: PowerLabs Technologies, Lagos, Nigeria

Abstract

Activation memory, not compute, limits CNN inference on constrained hardware such as microcontrollers. Direct in-place convolution removes the dual-buffer cost, but the memory-optimal formulation of Gural and Murmann assumes valid padding, unit stride, unit dilation, and odd square kernels, and needs a non-sequential traversal costing $2\times$ inference time in transposes. We identify two regimes in which their published closed form does not hold: (1) an under-allocation of exactly $(k-1)C_{in} \bmod (C_{out}-C_{in})$ scalars, active on every convolutional layer of their own deployed network and manifesting as a silent corruption of still-live input; (2) an unbounded overestimate, up to $2{,}432\times$ , once the critical leg leaves the output grid. We correct both and generalize to arbitrary stride, dilation, padding, and rectangular kernels. We then propose Right In-Place (RiP) convolution, a bit-identical operation in which every layer reads its input right-aligned in a shared workspace and writes its output left-aligned from index zero. The debt is piecewise affine in the output pixel index, so evaluating its breakpoints in $O(1)$ yields the minimum safe gap without enumerating the output grid, with row-major access preserved. Across $10{,}000$ random layers RiP produced no corruption, and across 84 convolutional layers from 25 architectures it matches the herringbone workspace exactly on 58 and within 5% on 81, using 24.8% less memory than dual buffering on average. Written into TinyEngine's kernels and deployed to a Raspberry Pi Pico 1 and Pico 2, it cuts peak activation memory across eleven MCUNet models by 12.5 to 33.3% at unchanged cycle counts and bit-identical outputs, raising the number of models that fit the Pico 1's 256 KB SRAM from six to nine.

Figures & tables

Figure 1 : Workspace layout on a Pico 1 for MCUNet-in3 under naive dual buffering, TinyEngine’s scheduler, and RiP. All three are placed by the same MILP memory allocator ( Sec. 4.3 ). On the first two it reaches the same peak as TinyEngine’s first-fit memory allocator [ 20 ] , so the allocator alone does not account for the saving; under in-place convolution first-fit fragments the arena and does not allow the full utilization of the extra memory, so the MILP is what lets RiP’s gap be reclaimed rather than stranded.

Method	Axis	General	Sequential	Bound	Restrictions
Herringbone [ 9 ]	spatial in-place	✓	$\times$	✓	valid, $s{=}1$ , $d{=}1$ , odd $k$ square
Replace [ 9 ]	spatial in-place	✓	✓	$\times$	valid, $s{=}1$ , $d{=}1$ , odd $k$ square
In-place DW [ 20 ]	channel in-place	$\times$	✓	–	depthwise only
MCUNetV2 [ 19 ]	cross-layer patch	✓	–	–	recomputation overhead
RAMAN [ 18 ]	hardware overlay	✓	–	$\times$	requires custom accelerator
CMSIS-NN Replace [ 25 ]	inter-layer arena	✓	✓	$\times$	whole-tensor granularity

Table 1: Memory-reduction axes for CNN inference on tiny devices. “General” = applies to standard channel-mixing convolutions; “Sequential” = preserves row-major output order without reordering; “Bound” = derives a closed-form minimum workspace for the layer.

Panel of Figure 2 — Figure 2 : Convolution traversal order.

$C_{in}$	$C_{out}$	$k$	$H{=}W$	$x^{*}$	G&M	Truth	Err.
Limitation 1: G&M’s deployed MNIST network, conv_0–conv_2
1	5	3	14	0.5	531	533	$-2$
5	8	3	12	3.3	148	149	$-1$
8	11	3	10	5.3	45	46	$-1$
Limitation 2: critical leg exceeds the output grid
3	4	5	8	12	79	7	$+72$

Table 2 : The two limitations in the as-published herringbone formula ( Eq. 1 ). Truth is the minimum corruption-free gap, found by binary search against the simulator; Err. is G&M minus Truth, so negative entries are under-allocations. Corollary 1 reproduces Truth on every row.

H{=}W

is the input spatial size.

Model	Buffer (B)	Base	RiP	Ratio	Base	RiP	Ratio	Base	RiP	Ratio	Acc.	Bit-exact
		Peak activation memory (B)			Pico 2 cycles (M)			Pico 1 time (ms)
MCUNet-in0	3,984	46,080	36,864	${4}{/5}$	39.5	39.7	$1.003$	1,772.4	1,772.0	$1.000$	$40.4\%$	✓
MCUNet-in1	5,864	92,160	73,728	${4}/{5}$	70.0	70.3	$1.004$	3,601.1	3,607.8	$1.002$	$49.9\%$	✓
MCUNet-in2	14,976	204,800	179,200	${7}/{8}$	307.0	307.2	$1.001$	15,978.1	15,999.5	$1.001$	$60.3\%$	✓
MCUNet-in3	18,536	247,808	185,856	${3}/{4}$	380.5	380.8	$1.001$	–	19,388.2	–	$61.8\%$	✓
MCUNet-in4	16,520	409,600	307,200	${3}/{4}$	708.2	708.7	$1.001$	–	–	–	$68.0\%$	✓

Table 3 : Eleven models from the MCUNet suite, baseline versus RiP. Buffer is the scratch and im2col allocation, which sits outside the arena and is unchanged by RiP. Peak activation memory is the arena alone, excluding that buffer, the SDK and TinyUSB .data / .bss , and the stack; it is a property of the allocation, so it is reported once for both boards. Cost is cycles count on the Pico 2 and wall-clock time on the Pico 1, whose RP2040 exposes no cycle counter. A dash marks a model the board could not run, since the arena is not the only claim on memory. Acc. is int8 top-1 as reported by [ 20 ] , identical under both strategies;

\dagger

marks Visual Wake Words rather than ImageNet, and

\ddagger

a single-class detector, for which no accuracy is reported. Bit-exact reports agreement of on-device output checksums against baseline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Convolution setup ( Sec. 4.1 )
$\mathbf{X},\mathbf{Y}$	Input, output activation tensors
$H_{in},W_{in},C_{in}$	Input height, width, channel count
$H_{out},W_{out},C_{out}$	Output height, width, channel count
$H_{p},W_{p}$	Padded input height, width
$N_{out}$	Output tensor size, $H_{out}W_{out}C_{out}$
$k_{h},k_{w}$	Kernel height, width

Appendix

Table 4 : Symbols used throughout the paper, grouped by the section that introduces them.

D

is overloaded by arity:

D(r,c)

is the row-major RiP debt of Eq. 7 ,

D(x)

the herringbone leg debt of Eq. 11 .

Panel of Figure 3 — Figure 3 : Herringbone traversal order on uneven output grids.

Tier	Generator	Share
1	plain convolution, $k\in[1,7]$	$11\%$
1	same padding, $p=\lfloor k/2\rfloor$	$8\%$
1	strided downsampling	$6\%$
1	inverted-bottleneck expansion	$5\%$
1	projection, $C_{out}\leq C_{in}$	$5\%$
2	collapsed minor axis, $n\in\{1,2,3\}$	$9\%$

Appendix

Table 5 : The twenty generators, with their share of the sweep. Tier 1 is the geometry shipping models are built from, tier 2, the small-grid and degenerate regimes where every defect found so far has lived, and tier 3, the correctness corners that generality requires but real models rarely reach.

Family	$n$	Gen.	Exact	${\leq}5\%$	Max	Save	Dense	Timed	Cost
ResNet-50	11	4	10	11	$1.006$	$25.5\%$	$2.3\%$	6	$1.000$
ResNet-56 (pruned)	5	5	4	5	$1.011$	$26.9\%$	$11.4\%$	5	$1.001$
VGG-16	4	4	0	4	$1.033$	$25.0\%$	$0.9\%$	–	–
VGG-16 (pruned)	3	3	0	1	$1.086$	$17.9\%$	$4.2\%$	3	$1.000$
MobileNetV1	6	1	5	6	$1.005$	$32.1\%$	$0.0\%$	3	$1.000$
MobileNetV2	5	1	4	5	$1.005$	$16.8\%$	$0.0\%$	2	$1.000$

Appendix

Table 6 : RiP against the generalized herringbone bound on 84 convolutional layers drawn from 25 architectures.

n

is layers sampled; Gen. counts those whose parameters fall outside the restrictions of [ 9 ] and so are reachable only through Theorem 11 . Exact and

{\leq}5\%

count layers where RiP’s workspace equals, or is within 5% of, herringbone’s. Max is the worst RiP-to-herringbone workspace ratio in the family. Save is the mean workspace reduction against dual buffering, and Dense the mean further reduction from not materializing padding ( Appendix F ). Timed counts the layers executed on the Pico 2 and Cost is their median runtime ratio against the dual-buffer kernel, right-shift included; a dash marks a family with no timed layer. Every layer passes the corruption simulator under both layouts, so no correctness column is shown.

Model	Ops	Res.	Vars	Constr.	Dual buf.	RiP	Ratio	Dual buf.	RiP	Ratio	In-place (%)	Shift (#)
			MILP size		Compile (s)			Peak activation (KB)			Tensor placement
MobileNet	29	0	1,361	1,969	0.08	0.14	1.69	1,176.0	784.0	2/3	60.0	–
MobileNetV2	64	10	6,364	8,813	0.26	0.52	2.01	1,470.0	1,176.0	4/5	49.2	–
MobileNetV3-S	109	15	18,201	24,834	1.08	1.49	1.37	392.0	220.5	9/16	64.5	–
MobileNetV3-L	122	18	22,751	30,967	1.27	1.42	1.12	980.0	784.0	4/5	82.1	–
VGG-16	21	0	729	1,077	0.11	0.16	1.49	6,272.0	3,150.1	0.502	54.5	1

Appendix

Table 7 : Graph-level allocation across 33 keras.applications models under the dense layout with the receptive field gathered first. Ops is graph nodes after folding and Res. the branch connections that keep two tensors alive at once. Vars and Constr. size the MILP; Compile is the full offline step (parse, model build, solve) under each formulation, median of 10 runs. Dual buffer is the same program restricted to disjoint placement, so it is the optimal packing of never-aliased tensors rather than a greedy baseline. In-place is the share of tensors the solver places in-place and Shift the number it separates by a physical right shift; a dash marks a model with no move scheduled. The memory ratio is an exact fraction on 29 of the 33, and the four exceptions are exactly the four models with a non-empty Shift column.

Explore similar work

Split CNN Inference on Networked Microcontrollers
May 10, 2026Junyu Lu, Shashwath Suresh, Hao Liu +2Neural Network Inference Cortex
Delft University of Technology, Delft, The Netherlands
Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency
Jul 2, 2026Tasnim ShahriarMobilenetv2 Neural Network Inference
Independent Researcher
Radio-Frequency Convolutional Neural Networks
Sep 16, 2026Zhihui Gao, Shi-Yuan Ma, Yiran Chen +2Edge Devices Neural Network Inference
Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA. · Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.

May 10, 2026, cs.DCJ/K move · Enter open · S save

Split CNN Inference on Networked Microcontrollers

Junyu Lu, Shashwath Suresh, Hao Liu, Qi Hong+1

Delft University of Technology, Delft, The Netherlands

Neural Network Inference Cortex Convolutional Neural Networks

Open paper AskPDF arXiv

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Abstract

Explore similar work

Split CNN Inference on Networked Microcontrollers

Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency

Radio-Frequency Convolutional Neural Networks