Beyond Convolution: A Taxonomy of Structured Operators for Learning-Based Image Processing

Simone Cammarasana

cs.CVMar 12, 2026

Beyond Convolution: A Taxonomy of Structured Operators for Learning-Based Image Processing

Authors: Simone Cammarasana

Organizations: CNR-IMATI, Via De Marini 6, Genova, Italy

Abstract

The convolution operator is the fundamental building block of modern convolutional neural networks (CNNs), owing to its simplicity, translational equivariance, and efficient implementation. However, its structure as a fixed, linear, locally-averaging operator limits its ability to capture structured signal properties such as low-rank decompositions, adaptive basis representations, and non-uniform spatial dependencies. This paper presents a systematic taxonomy of operators that extend or replace the standard convolution in learning-based image processing pipelines. We organise the landscape of alternative operators into five families: (i) decomposition-based operators, which separate structural and noise components through singular value or tensor decompositions; (ii) adaptive weighted operators, which modulate kernel contributions as a function of spatial position or signal content; (iii) basis-adaptive operators, which optimise the analysis bases together with the network weights; (iv) integral and kernel operators, which generalise the convolution to position-dependent and non-linear kernels; and (v) attention-based operators, which relax the locality assumption entirely. For each family, we provide a formal definition, a discussion of its structural properties with respect to the convolution, and a critical analysis of the tasks for which the operator is most appropriate. We further provide a comparative analysis of all families across relevant dimensions -- linearity, locality, equivariance, computational cost, and suitability for image-to-image and image-to-label tasks -- and outline the open challenges and future directions of this research area.

Figures & tables

Property	Convolution	Implication
Linearity	Yes	Efficient gradient; no local non-linearity
Transl. equivariance	Yes	Position-independent feature extraction
Locality	Yes (fixed $K$ )	Limited receptive field per layer
Uniform weighting	Yes	Content-agnostic neighbourhood averaging

Table 1: Structural properties of the standard convolution operator, used as a reference point for the taxonomy.

Operator family	Linear	Transl. equiv.	Local	Unif. weight.	Adapt. to content	Comp. cost	I2I	I2L	Key ref.
Standard convolution	✓	✓	✓	✓	$\times$	$\mathcal{O}(K^{2})$	✓	✓	[ 27 ]
Decomp.-based (SVD)	$\times$	$\times$	✓	$\times$	✓	$\mathcal{O}(K^{3})$	✓	$\circ$	[ 20 , 5 ]
Adapt. weighted (density)	✓	$\circ$	✓	$\times$	$\circ$	$\mathcal{O}(K^{2})$	✓	✓	[ 8 ]
Adapt. weighted (dynamic)	✓	✓	✓	$\times$	✓	$\mathcal{O}(K^{2})$	✓	✓	[ 13 ]
Basis-adaptive (F-transform)	✓	$\times$	$\times$	$\times$	✓	$\mathcal{O}(n\cdot\|\Omega\|)$	✓	$\circ$	[ 6 , 33 ]
Basis-adaptive (wavelet)	✓	$\times$	$\circ$	$\times$	$\circ$	$\mathcal{O}(N\log N)$	✓	$\circ$	[ 24 ]

Table 2: Comparison of operator families across structural properties and task suitability. I2I : image-to-image tasks (denoising, super-resolution). I2L : image-to-label tasks (classification, detection).

\circ

: partial; ✓: yes;

\times

: no.

Explore similar work

Aug 19, 2026

Multi-stage neural operator learning with application for convolutions

Convolution integrals widely exist in applications, and to enable fast and accurate computations, this paper introduces two general multi-stage neural operator learning frameworks. The first, Deep Collocation Neural Operator (DCNO), is a supervised approach that iteratively refines the operator approximation by learning residuals from input-output data pairs. The second, Deep Galerkin Neural Operator (DGNO), is an unsupervised framework applicable when the target operator can be represented by a PDE, leveraging the weak form of the PDE residual for training. Both methods progressively construct basis operators through multiple training stages to enrich the approximation space, leading to significantly improved accuracy over standard one-shot operator learning. We provide theoretical analysis for their approximation capabilities and implement them for learning convolutions. Extensive numerical experiments demonstrate that both DCNO and DGNO achieve high accuracy, approaching machine precision under single float for convolution problems, and offer substantial efficiency gains for numerous queries or parametric variations compared to traditional solvers. We also extend these frameworks to handle multi-input operator learning scenarios involving variations in both the density and kernel of a convolution.

Zhiping Mao, Zhenye Wen, Yong Zhang +1

School of Mathematical Sciences, Eastern Institute of Technology, Ningbo, 315200 Ningbo, China · School of Mathematics and Statistics, Wuhan University, 430072 Wuhan, China · Center for Applied Mathematics and KL-AAGDM, Tianjin University, 300072 Tianjin, China +2

Operator Learning Neural Operators PDE Operator Learning Neural Network Approximation Theory

Ask PDF

Nov 18, 2025

Unifying Convolution and Attention via Convolutional Nearest Neighbors

Convolutional Neural Networks and Vision Transformers are the two dominant architectural families in computer vision, defined by spatially local convolution and global self-attention respectively. Despite their apparent differences, we show that both operations are special cases of a single

k

-nearest neighbor aggregation framework: convolution selects neighbors by spatial proximity while attention selects by feature similarity, placing them at two ends of a shared operational spectrum. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that exactly recovers standard and depthwise convolution, self-attention, and sparse attention variants including KVT-attention as special cases, and exposes the design space of neighbor-selection strategies between them through configurable similarity functions, positional encodings, and aggregation kernels. We validate ConvNN on ImageNet-1K classification across two complementary architectures: a hybrid branching layer in ResNet-50 that combines local and global feature learning, improving top-1 accuracy by 3.0% over the ResNet-50 baseline, and ConvNN-attention in ViT-Base that achieves 81.64% top-1 accuracy, surpassing standard multi-head self-attention by 0.7%. Together, these results demonstrate that ConvNN provides a principled foundation for designing operations that bridge convolutional and attention-based computation.

Mingi Kang, Jeová Farias Sales Rocha Neto

Bowdoin College

Hybrid CNN-Transformer Architectures Self-Attention Vision Transformer Convolutional Neural Networks Attention Mechanisms k-Nearest Neighbors

Ask PDF

Sep 9, 2026

Muon-C: Operator-Aligned Muon for Convolutional Kernels

Muon replaces matrix momentum with an approximately orthogonal polar direction, but its geometry depends on the matrix representation. For convolution, standard unfolding describes a local patch map rather than the convolution operator. We introduce Muon-C, an operator-aligned optimizer that represents kernel momentum as frequency-wise channel-transfer matrices, polarizes these blocks independently, and uses a critical Fourier grid to return updates exactly to the original finite kernel support. We show that the new geometry arises from combining the block partition and Fourier coordinates. The exact-polar direction is a linear minimization oracle under the critically sampled convolution norm. Its worst-case guarantee relative to the continuous convolution-operator norm is never weaker than unfolding and is strictly stronger for

3\times3

kernels. On CIFAR-10 flow matching with matched applied-update RMS, Muon-C reaches 9.87 FID at 40k iterations, compared with 22.26 for unfolded Muon and 51.31 for Adam. It reaches their final quality using

0.62\times

and

0.64\times

their model FLOPs, respectively. Under equal tuning budgets, Muon-C achieves 3.42 FID. Gains persist across data scales and transfer to classification across convolutional architectures.

Jiaxin Qing, Lexin Li

University of California, Berkeley Berkeley, CA 94720-1776, USA

Neural Network Optimization Muon Optimizer

Ask PDF