Enabling Memory-efficient Im2win Convolution with Multi-precision Support on GPU CUDA and Tensor Cores

Xiang Fu; Jixiang Ma; Xinpeng Zhang; Peng Zhao; Shuai Lu; Xu Tony Liu

cs.DCAug 21, 2026

Enabling Memory-efficient Im2win Convolution with Multi-precision Support on GPU CUDA and Tensor Cores

Authors: Xiang Fu, Jixiang Ma, Xinpeng Zhang, Peng Zhao, Shuai Lu, Xu Tony Liu

Organizations: Nanchang Hangkong University, China · EEO Tech, China · University of Washington, USA · AWS, USA

Abstract

Convolution is a principal computational bottleneck in deep neural networks, and its efficiency depends on tight integration between algorithms and GPU hardware. Existing GPU convolution methods suffer from large memory overhead, poor cache utilization, limited effectiveness across kernel sizes, or numerical instability. This work extends the im2win paradigm -- a universal, memory-efficient convolution method with contiguous memory access for all kernel sizes -- to run efficiently in full precision on CUDA cores and half precision on tensor cores. By introducing new kernel designs and optimizations such as zig-zag memory access and asynchronous data movement, im2win efficiently exploits hardware-accelerated half-precision matrix multiply-accumulate operations. Across twelve CNN benchmarks, im2win achieves up to 2.8x higher TFLOPS than its CUDA core implementation, 1.4x higher than cuDNN, and 6.4x higher than GEMM-based convolution with cuBLAS, while using as little as 53% and 35% of their memory, respectively. These results establish im2win as a unified, high-performance convolution framework for modern GPU architectures.

Figures & tables

Fig. 1: The schematic diagram illustrates, from left to right, direct convolution, im2win convolution, and im2col-based convolution, separated by blue dashed lines. From top to bottom, these represent the input (1×3×3×3×3), filter (2×3×2×2), and output (1×2×2×2). Colors denote distinct channels, while arrows indicate transformation processes. Red, blue, and green boxes within the input highlight the input regions required to compute the first three matching output elements. These regions are multiplied by the filter and summed to yield the corresponding output elements.

NAME	INPUT	FILTER, STRIDE	OUTPUT
	$C_{i}\times H_{i}\times W_{i}$	$C_{o}\times H_{f}\times W_{f},s_{h}(s_{w})$	$C_{o}\times H_{o}\times W_{o}$
cv1	$3\times 227\times 227$	$96\times 11\times 11,4$	$96\times 55\times 55$
cv2	$3\times 231\times 231$	$96\times 11\times 11,4$	$96\times 56\times 56$
cv3	$3\times 227\times 227$	$64\times 7\times 7,2$	$64\times 111\times 111$
cv4	$64\times 224\times 224$	$64\times 7\times 7,2$	$64\times 109\times 109$
cv5	$96\times 24\times 24$	$256\times 5\times 5,1$	$256\times 20\times 20$

TABLE I: Twelve convolutional layers of the DNN benchmarks

NAME	CUDNN ALGO	CUDNN ALGO	IM2WIN ALGO
	CUDA CORES	TENSOR CORES	TENSOR CORES
cv1	IPG_CC	IPG_TC	baseline
cv2	IPG_CC	IPG_TC	wo_zigzag
cv3	IPG_CC	IPG_TC	baseline
cv4	FT_CC	IPG_TC	wo_async
cv5	WN_CC	IPG_TC	baseline

TABLE II: the observed best performant cuDNN convolutions and im2win in each benchmark.

Fig. 2: Performance results in TFLOPS and memory usage among the three convolutional algorithms (im2win, im2col_cuBLAS, and cuDNN) on CUDA and tensor cores respectively. Note: that the memory usage of im2col_cuBLAS_TC is 14.6 GB on cv3, and that of im2win_CC is 11.7 GB on cv4.

Fig. 3: Performance results in TFLOPS and the memory usage of the im2win, PyTorch’s im2col-based convolutions using cuBLAS, and cuDNN convolutions on GPU CUDA cores and tensor cores. Notes that some of these algorithms in cuDNN fail to run on some benchmarks due to certain constraints and TFLOPS of cuDNN_IPG_TC is 88.86 on cv5.

Fig. 4: Performance results in TFLOPS of the ablation study of three optimization techniques on tensor cores.

Explore similar work

Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference
Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient Inference Convolutional Neural Networks
PowerLabs Technologies, Lagos, Nigeria
WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution
Jul 2, 2026Wan Song, Wei Zhou, Rui Wang +4GPU Kernel Optimization Convolutional Neural Networks
Hefei University of Technology, Hefei, China · Lingyang Industrial Internet Co., Ltd., Hefei, China · Kochi University of Technology, Kochi, Japan.
Fast and Memory-Efficient Wavelet Convolutions via I/O-Aware Reformulation
Aug 11, 2026Amit Aflalo, Shahaf E. Finder, Roy Amoyal +2GPU Kernel Optimization Wavelet Methods
The Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel

Sep 30, 2026J/K move · Enter open · S save

Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye Eigbe

PowerLabs Technologies, Lagos, Nigeria

Memory-Efficient Inference Convolutional Neural Networks On-Device Inference

Open paper AskPDF arXiv

Activation memory, not compute, limits CNN inference on constrained hardware such as microcontrollers. Direct in-place convolution removes the dual-buffer cost, but the memory-optimal formulation of Gural and Murmann assumes valid padding, unit stride, unit dilation, and odd square kernels, and needs a non-sequential traversal costing

2\times

inference time in transposes. We identify two regimes in which their published closed form does not hold: (1) an under-allocation of exactly

(k-1)C_{in} \bmod (C_{out}-C_{in})

scalars, active on every convolutional layer of their own deployed network and manifesting as a silent corruption of still-live input; (2) an unbounded overestimate, up to

2{,}432\times

, once the critical leg leaves the output grid. We correct both and generalize to arbitrary stride, dilation, padding, and rectangular kernels. We then propose Right In-Place (RiP) convolution, a bit-identical operation in which every layer reads its input right-aligned in a shared workspace and writes its output left-aligned from index zero. The debt is piecewise affine in the output pixel index, so evaluating its breakpoints in

O(1)

yields the minimum safe gap without enumerating the output grid, with row-major access preserved. Across

10{,}000

random layers RiP produced no corruption, and across 84 convolutional layers from 25 architectures it matches the herringbone workspace exactly on 58 and within 5% on 81, using 24.8% less memory than dual buffering on average. Written into TinyEngine's kernels and deployed to a Raspberry Pi Pico 1 and Pico 2, it cuts peak activation memory across eleven MCUNet models by 12.5 to 33.3% at unchanged cycle counts and bit-identical outputs, raising the number of models that fit the Pico 1's 256 KB SRAM from six to nine.

Abstract

Explore similar work

Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution

Fast and Memory-Efficient Wavelet Convolutions via I/O-Aware Reformulation