Scale-invariant Gaussian derivative residual networks

Andrzej Perzanowski; Tony Lindeberg

Figure 1: Schematic illustrations of simplified residual blocks: (left) the simplest possible residual block of the form ( 1 ), based on a single convolution kernel

w(x)

, (right) the basic residual block of the form ( 2 ), based on two convolution kernels

w_{1}(x)

and

w_{2}(x)

. In both the diagrams, the skip connections perform identity mappings on the input

f(x)

, denoted as

id

.

Figure 2: Discretised Gaussian derivative kernels up to order 3, for (left) the scale parameter

\sigma=1

and for (right) the scale parameter

\sigma=4

. These receptive fields are obtained by applying central difference operators (corresponding to the given derivative order) to the discrete analogue of the Gaussian kernel, as defined in Equation ( 72 ). The ranges of both the axes of the filters are for (the left figure) in the interval

[-5,5]

, and for (the right figure) in the interval

[-16,16]

. (Top row) the zero-order Gaussian kernel

g(x_{1},x_{2};\;\sigma)

, (second row) first-order Gaussian derivatives

g_{x_{1}}(x_{1},x_{2};\;\sigma)

and

g_{x_{2}}(x_{1},x_{2};\;\sigma)

, (third row) second-order Gaussian derivatives

g_{x_{1}x_{1}}(x_{1},x_{2};\;\sigma)

,

g_{x_{1}x_{2}}(x_{1},x_{2};\;\sigma)

,

g_{x_{2}x_{2}}(x_{1},x_{2};\;\sigma)

, (bottom row) third-order Gaussian derivatives

g_{x_{1}x_{1}x_{1}}(x_{1},x_{2};\;\sigma)

,

g_{x_{1}x_{1}x_{2}}(x_{1},x_{2};\;\sigma)

,

g_{x_{1}x_{2}x_{2}}(x_{1},x_{2};\;\sigma)

and

g_{x_{2}x_{2}x_{2}}(x_{1},x_{2};\;\sigma)

.

Figure 3: Illustration of a single-scale-channel Gaussian derivative residual network consisting of 18 Gaussian derivative layers, together with the internal structure of a representative residual block, as well as an inside look into the composition of a Gaussian derivative layer within this residual block. The top part of the diagram illustrates the architecture of a single-scale-channel GaussDerResNet, with the layers

F_{\sigma_{0},k}

organised into residual blocks according to ( 22 ) and ( 35 , with the exception of the first and the last layers, which are regular Gaussian derivative layers according to ( 19 ), ( 34 ), and ( 36 ). Each residual block is constructed out of two Gaussian derivative layers together with a skip connection, and is shown in detail for the residual block

{\cal M}_{w_{14},w_{15},\sigma_{8}}

in the bottom right of the diagram, with the light grey dotted lines not part of the residual block and shown only for context. The pure single-scale-channel GaussDerResNet

\Lambda_{\sigma_{i,0}}^{c}(f)

applied the input image

f

according to ( 38 ) is parametrised by an initial scale value

\sigma_{i,0}

, with the scale parameter of each layer determined according to a geometric progression according to ( 37 ), resulting in the sizes of the receptive fields increasing with the network depth, with each residual block using the same scale parameter value for both of its layers. Furthermore, each Gaussian derivative layer in the network consists of a convolution with a linear combination of discretised Gaussian derivative basis kernels

w(x;\;\sigma)

according to ( 3 ), as visualised in the bottom left of the diagram for layer

14

, which is the first layer within the Gaussian derivative residual block at the level

k=8

. In the top row of that bottom-left figure, we see the contributions from the first order Gaussian derivatives, and in the bottom row the contributions from the second order Gaussian derivatives. Each basis kernel in the layer is computed by applying a corresponding central difference operator to a discrete analogue of the Gaussian kernel according to ( 72 ). The parameters

\tilde{C_{\alpha}}=m(\alpha)\,C_{\alpha}\,\sigma^{|\alpha|}

represent the learned weights of the layer, with the tilde symbol over the weight parameter

C_{\alpha}

including the effects of the corresponding scale normalisation factor

\sigma^{|\alpha|}

and the multinomial normalisation factor

m(\alpha)

, both defined according ( 3 ) and ( 68 ). Finally, the output of the single-scale-channel GaussDerResNet is obtained by applying a spatial selection stage

\Gamma_{\sigma_{i,0}}^{c}(f)=\operatorname{SpatSel}(\Lambda_{\sigma_{i,0}}^{c}(f))

according to ( 41 ) to the output of the final layer

F_{\sigma_{0},K}

.

Figure 4: Conceptual illustration of a multi-scale-channel Gaussian derivative residual network, composed of 4 parallel single-scale-channel GaussDerResNets, referred to as scale channels. Each scale channel is constructed as shown in the top part of Figure 3 , consisting of the (i) convolutional stage

\Lambda_{\sigma_{i,0}}^{c}

, composed of 18 Gaussian derivative layers, represented by the conical frustums (meaning the tapered cylinders) in the diagram, indicating the increase of receptive field size with depth, and (ii) the spatial selection stage. There is a geometric distribution of the spacing between the initial scale values

\sigma_{i,0}

, that the scale channels are based on, set by a fixed ratio, as expressed in Equation ( 46 ). Each scale channel processes a copy of the input image, and crucially, in order to achieve scale covariance, the scale channels share the same weights, with all the scale channels parametrised by the same relative scale ratio

r

. Finally, instead of a fully connected classification layer, permutation-invariant pooling over the scale channels is performed as a final processing stage in the network.

\begin{CD}f^{\prime}@>{\mathcal{G}_{\sigma^{\prime}_{1}}}>{}>F_{1}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{2}}}>{}>F_{2}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{3}}}>{}>F_{3}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{4}}}>{}>F_{4}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{5}}}>{}>F_{5}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{6}}}>{}>F_{6}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{7}}}>{}>F_{7}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{8}}}>{}>F_{8}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{9}}}>{}>F_{9}^{\prime}@>{\mathcal{G}_{\sigma^{\prime}_{10}}}>{}>F_{10}^{\prime}\\ @A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A\\ f@>{\mathcal{G}_{\sigma_{1}}}>{}>F_{1}@>{\mathcal{M}_{\sigma_{2}}}>{}>F_{2}@>{\mathcal{M}_{\sigma_{3}}}>{}>F_{3}@>{\mathcal{M}_{\sigma_{4}}}>{}>F_{4}@>{\mathcal{M}_{\sigma_{5}}}>{}>F_{5}@>{\mathcal{M}_{\sigma_{6}}}>{}>F_{6}@>{\mathcal{M}_{\sigma_{7}}}>{}>F_{7}@>{\mathcal{M}_{\sigma_{8}}}>{}>F_{8}@>{\mathcal{M}_{\sigma_{9}}}>{}>F_{9}@>{\mathcal{G}_{\sigma_{10}}}>{}>F_{10}\end{CD}

Figure 5: Commutative diagram for the entire 18-layer Gaussian derivative residual network, illustrating the scale-covariant properties of the architecture. In the diagram,

\mathcal{G}_{\sigma_{k}}=\mathcal{G}_{w_{k},\sigma_{k}}

represents Gaussian derivative layer at the scale

\sigma_{k}

according to ( 19 ),

\mathcal{M}_{\sigma_{k}}=\mathcal{M}_{w_{2k-2,2k-1,\sigma_{k}}}

represents a standard Gaussian derivative residual block at the scale

\sigma_{k}

according to ( 22 ) and ( 35 ), where both there operators are based on Gaussian derivative primitives up to a given order

N

of spatial differentiation according to ( 3 ). The scaling operator

\mathcal{S}

represents a uniform scaling transformation with a scaling factor

S_{x}\in\mathbb{R}_{+}

of the form

x^{\prime}=S_{x}\,x

, with the associated transformation of the scale parameters according to

\sigma^{\prime}=S_{x}\,\sigma

. This scaling operator acts on both the input image

f

or any layer output

F_{k}=F_{\sigma_{0},k}

from the residual blocks, according to

f^{\prime}(x^{\prime})=f(x)

and

F^{\prime}_{\sigma^{\prime}_{0},k}(x^{\prime})=F_{\sigma_{0},k}(x)

for

\sigma^{\prime}_{0}=S_{x}\,\sigma_{0}

, leading to the abbreviated notation

F^{\prime}_{k}(x^{\prime})

=

{F}_{k}(x)

above. The commutative diagram should be read from the bottom left to the top right, and shows that each level in the cascade of the network can be matched under arbitrary scaling transformations

{\cal S}

, provided that such matching has also been done in the same manner at every previous step in the hierarchy, including the input image. As described in Section 3.1 , these scale covariance properties are not affected by the use of residual connections, or the presence non-linear

\operatorname{ReLU}

or batch normalisation stages in the network.

Figure 6: A collage made up of a representative subset of images from the rescaled STL-10 dataset. The (top row) contains samples from the training set (size factor 1), each with

96\times 96

pixels (with the pixel size adapted to fit the collage), with one example per class, where the 10 classes are: “airplane”, “bird”, “car”, “cat”, “deer”, “dog”, “horse”, “monkey”, “ship” and “truck”. In a similar manner, the (middle row) depicts samples from the size factor 2 test set, each with

192\times 192

pixels. The (bottom row) depicts a single sample from the “cat” class in the test set, for each possible scaling factor between 1/2 and 2, with the scaling factor increasing from left to right, and the images at all the size factors extended (if needed) using mirroring to the final image size of

192\times 192

pixels. Note that this means that the size factor 1 test set has mirror extensions and a larger image size compared to the training set.

Model Parameters	Symbol	rescaled Fashion-MNIST	rescaled CIFAR-10	rescaled STL-10
Number of scale channels	$N$	6	6	6
Initial scale value range	[ $\sigma_{i,0}$ ]	[ $\sqrt{2}^{-3},2$ ]	[ $\sqrt{2}^{-3},2$ ]	[ $\sqrt{2}^{-3},2$ ]
Relative scale ratio	$r$	1.13	1.2	1.32
Relative truncation error bound	$\epsilon$	0.01	0.005	0.005
Number of layers	$K$	18	18	18
Intermediate feature channels	$c$	1-48-24- $\overline{32}$ - $\overline{48}$ - $\overline{64}$ -128-10	3- $\overline{48}$ - $\overline{\overline{64}}$ - $\overline{\overline{192}}$ -256-10	3- $\overline{48}$ - $\overline{64}$ - $\overline{128}$ - $\overline{192}$ -256-10

Table 1: Key parameter values and mechanisms employed in our experiments with the multi-scale-channel GaussDerResNets, for the three rescaled datasets. In the row specifying the number of intermediate feature channels

c

, a line over a number means that two blocks in a row use the same value, while two lines means that three blocks in a row use the same value.

Training Parameters	rescaled Fashion-MNIST	rescaled CIFAR-10	rescaled STL-10
Number of training iterations (epochs)	30000 (32)	100000 (90)	56000 (119)
Batch size	64	45	17
Initial learning rate	0.01	0.01	0.01
Number of warm-up epochs	0	8	8
Weight decay	0.025	0.025	0.025
Scale channel dropout factor	0.2	0.3	0.3

Table 2: Training configurations for multi-scale-channel GaussDerResNet employed in our experiments, for the three rescaled datasets.

Network	#Params	Test accuracy
GaussDerResNet (no LS)	2.1M	$88.05\%$
WideResNet	11M	$88.52\%$
GaussDerResNet (with LS)	2.1M	$89.36\%$
Harm-WideResNet	11M	$90.45\%$
SESN-B	11M	$91.49\%$

Table 3: Comparison of the test accuracies and the numbers of parameters of networks trained on the original STL-10 dataset, showing the performance obtained by (i) the single-scale-channel GaussDerResNet with zero-order terms trained without label smoothing (abbreviated as “no LS”), (ii) the single-scale-channel GaussDerResNet with zero-order terms trained with label smoothing (abbreviated “with LS”), (iii) the Wide Residual (WideResNet) network (Zagoruyko and Komodakis 2016 ), (iv) the Harmonic Wide Residual (Harm WideResNet) network (Ulicny et al. 2019 ), and (v) the scale-equivariant steerable (SESN-B) network (Sosnovik et al. 2020 ). We can see that GaussDerResNets can achieve accuracies comparable to the other well performing networks, though marginally lower to the best performing one, while making use of far fewer network parameters.

Figure 7: Comparison between the scale generalisation curves for the rescaled Fashion-MNIST dataset, showing both the mean and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale, also with comparisons to single run of multi-scale-channel GaussDerNets based on max, logsumexp or average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). For the test data with the same size factor 1 as the training data, represented by the shaded grey region, the GaussDerResNets achieved a mean accuracy of 94.3% (with standard deviation

\pm

0.2%), which is

\sim

1.5 ppt higher compared to the GaussDerNets. From the graphs, we can see that the GaussDerResNets have overall better performance compared to the GaussDerNets, while retaining very good scale generalisation, in agreement with the theoretical analysis.

Figure 8: Comparison between the scale generalisation curves for the rescaled CIFAR-10 dataset, showing both the mean accuracy and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale, also with comparisons to single run of multi-scale-channel GaussDerNets based on max, logsumexp or average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). Each network was trained on training data for the size factor 1, and then evaluated on test data covering all the size factors between 1/2 and 2, with each copy of the test set only containing images at a single given scale. For the test data with the same size factor 1 as the training data, represented by the shaded grey region, the GaussDerResNets achieved a mean accuracy of 92.2-92.5% (with standard deviation

\pm

0.15-0.21%), which is

\sim

7 ppt higher compared to the GaussDerNets. From the graphs, we can see that the scale generalisation properties of the GaussDerResNets is significantly better compared to the GaussDerNets.

Figure 9: Scale generalisation curves for the (a) traffic sign and (b) aerial image subsets of the STIR dataset, comparing both the mean accuracy and the error bars for the standard deviation over 5 runs for the multi-scale-channel GaussDerResNets based on logsumexp pooling over scale to the multi-scale-channel GaussDerNets based on average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). Three different training regimes are shown for each dataset, indicated by shaded grey regions, with the networks trained on data containing (left) small size objects (size

\in[17,32]

), (middle) medium size objects (size

\in[33,48]

), and (right) large size objects (size

\in[49,64]

). All the networks were evaluated on the test set containing all the possible object sizes

l\in[17,64]

. From the graphs, we see that the GaussDerResNets have better scale generalisation properties than the GaussDerNets. (Note that the vertical axes for the test accuracy in the top and bottom subfigures use different lower bounds.)

Figure 10: Comparison between the scale generalisation curves for the rescaled STL-10 dataset, showing both the mean accuracy and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale. In this experiment, the networks were trained on training data corresponding to the size factor 1, and were then evaluated on rescaled copies of the test set, covering all the size factors between 1/2 and 2, with each copy corresponding to a different single size factor. The multi-scale-channel networks achieved a mean accuracy of 88.2–89.5% (with standard deviation

\pm

0.7-1.4%) on the test data of the same size (the size factor 1) as the training data, represented by the shaded grey region in the graph. As can be seen in the figure, despite a portion of the STL-10 images containing non-centered objects, the multi-scale-channel GaussDerResNets based on the spatial max pooling mechanism lead to rather flat scale generalisation curves.

Figure 11: Scale selection histograms averaged over 5 runs for the multi-scale-channel GaussDerResNets trained and evaluated on the rescaled CIFAR-10 dataset, with no label smoothing used during training, for the three different permutation invariant aggregation methods for pooling over scales, including (a) max pooling, (b) logsumexp pooling and (c) average pooling. In these scale selection histograms, the vertical axis represents the relative contribution of each scale channel to the final classification as function of the different size factors for the test dataset as mapped onto the horizontal axis. A clear linear trend can be seen in all the three histograms, with the dominant selected scale levels, represented by the dark regions, found to be directly proportional to the size factor of the test images. This behaviour is an explicit manifestation of the scale-covariant properties of the GaussDerResNet architecture.

Figure 13: Comparison between the scale generalisation curves and the corresponding scale selection histograms for the multi-scale-channel GaussDerResNets, averaged across 5 runs, based on either standard 2-jet layers or 2-jet layers complemented with a zero-order term, on the rescaled (left) Fashion-MNIST, (middle) CIFAR-10 and (right) STL-10 datasets. The subfigures use networks based on logsumexp pooling over scales. After being trained on the training set for the size factor 1, the networks were tested on all the size factors between 1/2 and 2. The scale selection histograms visualise the relative contribution of each scale channel to the final classification. For the rescaled Fashion-MNIST and CIFAR-10 datasets, an extra boundary scale channel

\sigma_{0}=2^{5/4}

was added during inference, although we do not show its contribution in the histograms for the sake of simplicity. As can be seen from the graphs, including a zeroth-order term clearly improves both the accuracy and the scale generalisation for the rescaled STL-10 dataset. For the rescaled Fashion-MNIST and CIFAR-10 datasets, using a zeroth-order term does not lead to any improvement, with the performance at the smaller size factors noticeably reduced when including this term. Additionally, a clear linear trend can be seen in the scale selection histograms for all the three datasets, demonstrating that including a zero-order term in the computational primitives of the GaussDerResNets leads to the selected scale levels still being proportional to the size in the test image data, as expected from theory. (Note that the vertical axes for the test accuracy in the top subfigures use different lower bounds for the different datasets.)

Figure 14: Activation maps of correctly classified images from the rescaled STL-10 dataset, extracted from the final layer of a multi-scale-channel GaussDerResNet with a spatial selection stage based on spatial max pooling and scale selection stage based on average pooling over scales. The activation maps reveal which spatial areas within the image elicit strong responses at the feature channel corresponding to the correct class, for scale channels based on initial scale values

\sigma_{0}\in\{1/2,1,2\}

, given test samples chosen from size factor 2 test set. Each row depicts results from one of the five chosen classes: (a) “bird”, (b) “car”, (c) “dog”, (d) “monkey” and (e) “ship”. As can be seen from the subfigures, the network typically makes a prediction by identifying characteristic parts of objects, either at fine or coarse scales, which correspond to strong responses at finer scale channels or coarser scale channels, respectively.

Figure 15: Illustration of the main steps in the generation of object hypotheses from deep scale-space maxima extracted from the deep scale-space representations for each class, for the given multi-object test image generated out of Fashion-MNIST samples of different sizes. (a) Original image. (b) Circle representations of the 10 strongest deep scale-space maxima, with the radius proportional to the selected scale for each deep scale-space maximum, and using a relative post-smoothing scale of

\gamma_{\text{tot}}=0.2

before the detection of deep scale-space maxima. (c) The result of suppressing overlapping spatial representations within each class. (d) The result of suppressing overlapping spatial representations across all the classes, preserving only the spatial representation having the strongest magnitude value, purely for visualisation purposes. From the resulting circles we can see that the network produces maxima over the objects, no matter the object scale. The colours of the circles indicate the classes of the object hypotheses, with red = “bag” , green = “trouser” , blue = “sneaker” and purple for all the other classes . The underlying image data in (b)–(d) have been brightened to improve clarity.

Figure 16: Illustration of the scale-covariant and scale-invariant properties of the resulting method for generating object hypotheses from deep scale-space maxima from the deep scale-space representation obtained by combining the final layers for the different values of the initial scale parameter for each class. In the top row, images of three objects from the Fashion-MNIST test set have been put at random positions for the size factor 1. In the bottom row, the same objects have been put at other random positions for the size factor 2. As can be seen from the illustrations, the sizes of the circles largely follow the amount of spatial scaling of the input data by the factor of 2 between the first and the second rows. The colours of the circles indicate the classes of the object hypotheses, with red = “ankle boot” , green = “coat” , blue = “bag” and purple for all the other classes . With the exception of the leftmost images, the underlying image data have been brightened to improve clarity.

Figure 17: Illustration of the main steps in the generation of object hypotheses, using three images from the COCO dataset, from deep scale-space maxima obtained by applying a GaussDerResNet based on max pooling over scales, trained on the rescaled STL-10 dataset. Subfigures (a) and (c) both show results for images with a cat and dog seated next each other, while subfigure (b) shows a cat in front of a car. From the resulting circles, we can see that that the network is able to produce strong maxima over multiple objects within real-world images, with the correct classes being the most dominant over the corresponding objects. The colours of the circles indicate the classes of the object hypotheses, with red = “car” , green = “dog” , blue = “cat” and purple for all the other classes .

	rescaled Fashion-MNIST	rescaled CIFAR-10	rescaled STL-10
# network parameters (GaussDerResNets)	295k	1.78M	2.07M
# network parameters (DSGaussDerResNets)	120k	547k	669k

Scale-invariant Gaussian derivative residual networks

Abstract

Figures & tables

Appendix

Explore similar work

Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets

A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks

FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution