cs.CVMar 3, 2026
SaveScale-invariant Gaussian derivative residual networks
Organizations: Computational Brain Science Lab, Department of Computational Science and Technology, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden.
Abstract
Generalisation across image scales remains a fundamental challenge for deep networks, which often fail to handle images at scales not seen during training (the out-of-distribution problem). In this paper, we present provably scale-invariant Gaussian derivative residual networks (GaussDerResNets), constructed out of scale-covariant Gaussian derivative residual blocks coupled in cascade, aimed at addressing this problem. By adding residual skip connections to the previous notion of Gaussian derivative layers, deeper networks with substantially increased accuracy can be constructed, while preserving very good scale generalisation properties. Explicit proofs are provided for the underlying scale-covariant and scale-invariant properties in arbitrary dimensions. To analyse the ability of GaussDerResNets to generalise to new scales, we apply them on a new rescaled version of the STL-10 dataset, where training is done at a single fixed scale and evaluation is performed on copies of the test set, each rescaled to a distinct spatial scale, with scale factors extending over a range of 4. We also conduct similar systematic experiments on the rescaled versions of Fashion-MNIST and CIFAR-10 datasets, and the existing STIR datasets. Experimentally, we demonstrate that the GaussDerResNets have strong scale generalisation and scale selection properties on all the four considered datasets with scaling variations. In our ablation studies, we investigate different architectural variants of GaussDerResNets, demonstrating that basing the architecture on depthwise-separable convolutions reduces the number of parameters and computations, with reasonably maintained accuracy and scale generalisation. We conclude by outlining how the proposed GaussDerResNets can be extended to joint local spatial and scale selection, to address the topic of multi-object detection in a provably scale-invariant manner.
Figures & tables
Figure 1: Schematic illustrations of simplified residual blocks: (left) the simplest possible residual block of the form ( 1 ), based on a single convolution kernel , (right) the basic residual block of the form ( 2 ), based on two convolution kernels and . In both the diagrams, the skip connections perform identity mappings on the input , denoted as .
Figure 2: Discretised Gaussian derivative kernels up to order 3, for (left) the scale parameter and for (right) the scale parameter . These receptive fields are obtained by applying central difference operators (corresponding to the given derivative order) to the discrete analogue of the Gaussian kernel, as defined in Equation ( 72 ). The ranges of both the axes of the filters are for (the left figure) in the interval , and for (the right figure) in the interval . (Top row) the zero-order Gaussian kernel , (second row) first-order Gaussian derivatives and , (third row) second-order Gaussian derivatives , , , (bottom row) third-order Gaussian derivatives , , and .
Figure 3: Illustration of a single-scale-channel Gaussian derivative residual network consisting of 18 Gaussian derivative layers, together with the internal structure of a representative residual block, as well as an inside look into the composition of a Gaussian derivative layer within this residual block. The top part of the diagram illustrates the architecture of a single-scale-channel GaussDerResNet, with the layers organised into residual blocks according to ( 22 ) and ( 35 , with the exception of the first and the last layers, which are regular Gaussian derivative layers according to ( 19 ), ( 34 ), and ( 36 ). Each residual block is constructed out of two Gaussian derivative layers together with a skip connection, and is shown in detail for the residual block in the bottom right of the diagram, with the light grey dotted lines not part of the residual block and shown only for context. The pure single-scale-channel GaussDerResNet applied the input image according to ( 38 ) is parametrised by an initial scale value , with the scale parameter of each layer determined according to a geometric progression according to ( 37 ), resulting in the sizes of the receptive fields increasing with the network depth, with each residual block using the same scale parameter value for both of its layers. Furthermore, each Gaussian derivative layer in the network consists of a convolution with a linear combination of discretised Gaussian derivative basis kernels according to ( 3 ), as visualised in the bottom left of the diagram for layer , which is the first layer within the Gaussian derivative residual block at the level . In the top row of that bottom-left figure, we see the contributions from the first order Gaussian derivatives, and in the bottom row the contributions from the second order Gaussian derivatives. Each basis kernel in the layer is computed by applying a corresponding central difference operator to a discrete analogue of the Gaussian kernel according to ( 72 ). The parameters represent the learned weights of the layer, with the tilde symbol over the weight parameter including the effects of the corresponding scale normalisation factor and the multinomial normalisation factor , both defined according ( 3 ) and ( 68 ). Finally, the output of the single-scale-channel GaussDerResNet is obtained by applying a spatial selection stage according to ( 41 ) to the output of the final layer .
Figure 4: Conceptual illustration of a multi-scale-channel Gaussian derivative residual network, composed of 4 parallel single-scale-channel GaussDerResNets, referred to as scale channels. Each scale channel is constructed as shown in the top part of Figure 3 , consisting of the (i) convolutional stage , composed of 18 Gaussian derivative layers, represented by the conical frustums (meaning the tapered cylinders) in the diagram, indicating the increase of receptive field size with depth, and (ii) the spatial selection stage. There is a geometric distribution of the spacing between the initial scale values , that the scale channels are based on, set by a fixed ratio, as expressed in Equation ( 46 ). Each scale channel processes a copy of the input image, and crucially, in order to achieve scale covariance, the scale channels share the same weights, with all the scale channels parametrised by the same relative scale ratio . Finally, instead of a fully connected classification layer, permutation-invariant pooling over the scale channels is performed as a final processing stage in the network.
| \begin{CD}f^{\prime}@>{\mathcal{G}_{\sigma^{\prime}_{1}}}>{}>F_{1}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{2}}}>{}>F_{2}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{3}}}>{}>F_{3}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{4}}}>{}>F_{4}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{5}}}>{}>F_{5}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{6}}}>{}>F_{6}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{7}}}>{}>F_{7}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{8}}}>{}>F_{8}^{\prime}@>{\mathcal{M}_{\sigma^{\prime}_{9}}}>{}>F_{9}^{\prime}@>{\mathcal{G}_{\sigma^{\prime}_{10}}}>{}>F_{10}^{\prime}\\ @A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A@A{}A{\mathcal{S}}A\\ f@>{\mathcal{G}_{\sigma_{1}}}>{}>F_{1}@>{\mathcal{M}_{\sigma_{2}}}>{}>F_{2}@>{\mathcal{M}_{\sigma_{3}}}>{}>F_{3}@>{\mathcal{M}_{\sigma_{4}}}>{}>F_{4}@>{\mathcal{M}_{\sigma_{5}}}>{}>F_{5}@>{\mathcal{M}_{\sigma_{6}}}>{}>F_{6}@>{\mathcal{M}_{\sigma_{7}}}>{}>F_{7}@>{\mathcal{M}_{\sigma_{8}}}>{}>F_{8}@>{\mathcal{M}_{\sigma_{9}}}>{}>F_{9}@>{\mathcal{G}_{\sigma_{10}}}>{}>F_{10}\end{CD} |
Figure 5: Commutative diagram for the entire 18-layer Gaussian derivative residual network, illustrating the scale-covariant properties of the architecture. In the diagram, represents Gaussian derivative layer at the scale according to ( 19 ), represents a standard Gaussian derivative residual block at the scale according to ( 22 ) and ( 35 ), where both there operators are based on Gaussian derivative primitives up to a given order of spatial differentiation according to ( 3 ). The scaling operator represents a uniform scaling transformation with a scaling factor of the form , with the associated transformation of the scale parameters according to . This scaling operator acts on both the input image or any layer output from the residual blocks, according to and for , leading to the abbreviated notation = above. The commutative diagram should be read from the bottom left to the top right, and shows that each level in the cascade of the network can be matched under arbitrary scaling transformations , provided that such matching has also been done in the same manner at every previous step in the hierarchy, including the input image. As described in Section 3.1 , these scale covariance properties are not affected by the use of residual connections, or the presence non-linear or batch normalisation stages in the network.
Figure 6: A collage made up of a representative subset of images from the rescaled STL-10 dataset. The (top row) contains samples from the training set (size factor 1), each with pixels (with the pixel size adapted to fit the collage), with one example per class, where the 10 classes are: “airplane”, “bird”, “car”, “cat”, “deer”, “dog”, “horse”, “monkey”, “ship” and “truck”. In a similar manner, the (middle row) depicts samples from the size factor 2 test set, each with pixels. The (bottom row) depicts a single sample from the “cat” class in the test set, for each possible scaling factor between 1/2 and 2, with the scaling factor increasing from left to right, and the images at all the size factors extended (if needed) using mirroring to the final image size of pixels. Note that this means that the size factor 1 test set has mirror extensions and a larger image size compared to the training set.
| Model Parameters | Symbol | rescaled Fashion-MNIST | rescaled CIFAR-10 | rescaled STL-10 |
|---|---|---|---|---|
| Number of scale channels | 6 | 6 | 6 | |
| Initial scale value range | [ ] | [ ] | [ ] | [ ] |
| Relative scale ratio | 1.13 | 1.2 | 1.32 | |
| Relative truncation error bound | 0.01 | 0.005 | 0.005 | |
| Number of layers | 18 | 18 | 18 | |
| Intermediate feature channels | 1-48-24- - - -128-10 | 3- - - -256-10 | 3- - - - -256-10 |
Table 1: Key parameter values and mechanisms employed in our experiments with the multi-scale-channel GaussDerResNets, for the three rescaled datasets. In the row specifying the number of intermediate feature channels , a line over a number means that two blocks in a row use the same value, while two lines means that three blocks in a row use the same value.
| Training Parameters | rescaled Fashion-MNIST | rescaled CIFAR-10 | rescaled STL-10 |
|---|---|---|---|
| Number of training iterations (epochs) | 30000 (32) | 100000 (90) | 56000 (119) |
| Batch size | 64 | 45 | 17 |
| Initial learning rate | 0.01 | 0.01 | 0.01 |
| Number of warm-up epochs | 0 | 8 | 8 |
| Weight decay | 0.025 | 0.025 | 0.025 |
| Scale channel dropout factor | 0.2 | 0.3 | 0.3 |
Table 2: Training configurations for multi-scale-channel GaussDerResNet employed in our experiments, for the three rescaled datasets.
| Network | #Params | Test accuracy |
|---|---|---|
| GaussDerResNet (no LS) | 2.1M | |
| WideResNet | 11M | |
| GaussDerResNet (with LS) | 2.1M | |
| Harm-WideResNet | 11M | |
| SESN-B | 11M |
Table 3: Comparison of the test accuracies and the numbers of parameters of networks trained on the original STL-10 dataset, showing the performance obtained by (i) the single-scale-channel GaussDerResNet with zero-order terms trained without label smoothing (abbreviated as “no LS”), (ii) the single-scale-channel GaussDerResNet with zero-order terms trained with label smoothing (abbreviated “with LS”), (iii) the Wide Residual (WideResNet) network (Zagoruyko and Komodakis 2016 ), (iv) the Harmonic Wide Residual (Harm WideResNet) network (Ulicny et al. 2019 ), and (v) the scale-equivariant steerable (SESN-B) network (Sosnovik et al. 2020 ). We can see that GaussDerResNets can achieve accuracies comparable to the other well performing networks, though marginally lower to the best performing one, while making use of far fewer network parameters.
Figure 7: Comparison between the scale generalisation curves for the rescaled Fashion-MNIST dataset, showing both the mean and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale, also with comparisons to single run of multi-scale-channel GaussDerNets based on max, logsumexp or average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). For the test data with the same size factor 1 as the training data, represented by the shaded grey region, the GaussDerResNets achieved a mean accuracy of 94.3% (with standard deviation 0.2%), which is 1.5 ppt higher compared to the GaussDerNets. From the graphs, we can see that the GaussDerResNets have overall better performance compared to the GaussDerNets, while retaining very good scale generalisation, in agreement with the theoretical analysis.
Figure 8: Comparison between the scale generalisation curves for the rescaled CIFAR-10 dataset, showing both the mean accuracy and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale, also with comparisons to single run of multi-scale-channel GaussDerNets based on max, logsumexp or average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). Each network was trained on training data for the size factor 1, and then evaluated on test data covering all the size factors between 1/2 and 2, with each copy of the test set only containing images at a single given scale. For the test data with the same size factor 1 as the training data, represented by the shaded grey region, the GaussDerResNets achieved a mean accuracy of 92.2-92.5% (with standard deviation 0.15-0.21%), which is 7 ppt higher compared to the GaussDerNets. From the graphs, we can see that the scale generalisation properties of the GaussDerResNets is significantly better compared to the GaussDerNets.
Figure 9: Scale generalisation curves for the (a) traffic sign and (b) aerial image subsets of the STIR dataset, comparing both the mean accuracy and the error bars for the standard deviation over 5 runs for the multi-scale-channel GaussDerResNets based on logsumexp pooling over scale to the multi-scale-channel GaussDerNets based on average pooling over scale (reproduced from Perzanowski and Lindeberg ( 2025d )). Three different training regimes are shown for each dataset, indicated by shaded grey regions, with the networks trained on data containing (left) small size objects (size ), (middle) medium size objects (size ), and (right) large size objects (size ). All the networks were evaluated on the test set containing all the possible object sizes . From the graphs, we see that the GaussDerResNets have better scale generalisation properties than the GaussDerNets. (Note that the vertical axes for the test accuracy in the top and bottom subfigures use different lower bounds.)
Figure 10: Comparison between the scale generalisation curves for the rescaled STL-10 dataset, showing both the mean accuracy and the error bars for the standard deviation over 5 runs for multi-scale-channel GaussDerResNets based on either max, logsumexp or average pooling over scale. In this experiment, the networks were trained on training data corresponding to the size factor 1, and were then evaluated on rescaled copies of the test set, covering all the size factors between 1/2 and 2, with each copy corresponding to a different single size factor. The multi-scale-channel networks achieved a mean accuracy of 88.2–89.5% (with standard deviation 0.7-1.4%) on the test data of the same size (the size factor 1) as the training data, represented by the shaded grey region in the graph. As can be seen in the figure, despite a portion of the STL-10 images containing non-centered objects, the multi-scale-channel GaussDerResNets based on the spatial max pooling mechanism lead to rather flat scale generalisation curves.
Figure 11: Scale selection histograms averaged over 5 runs for the multi-scale-channel GaussDerResNets trained and evaluated on the rescaled CIFAR-10 dataset, with no label smoothing used during training, for the three different permutation invariant aggregation methods for pooling over scales, including (a) max pooling, (b) logsumexp pooling and (c) average pooling. In these scale selection histograms, the vertical axis represents the relative contribution of each scale channel to the final classification as function of the different size factors for the test dataset as mapped onto the horizontal axis. A clear linear trend can be seen in all the three histograms, with the dominant selected scale levels, represented by the dark regions, found to be directly proportional to the size factor of the test images. This behaviour is an explicit manifestation of the scale-covariant properties of the GaussDerResNet architecture.
Figure 13: Comparison between the scale generalisation curves and the corresponding scale selection histograms for the multi-scale-channel GaussDerResNets, averaged across 5 runs, based on either standard 2-jet layers or 2-jet layers complemented with a zero-order term, on the rescaled (left) Fashion-MNIST, (middle) CIFAR-10 and (right) STL-10 datasets. The subfigures use networks based on logsumexp pooling over scales. After being trained on the training set for the size factor 1, the networks were tested on all the size factors between 1/2 and 2. The scale selection histograms visualise the relative contribution of each scale channel to the final classification. For the rescaled Fashion-MNIST and CIFAR-10 datasets, an extra boundary scale channel was added during inference, although we do not show its contribution in the histograms for the sake of simplicity. As can be seen from the graphs, including a zeroth-order term clearly improves both the accuracy and the scale generalisation for the rescaled STL-10 dataset. For the rescaled Fashion-MNIST and CIFAR-10 datasets, using a zeroth-order term does not lead to any improvement, with the performance at the smaller size factors noticeably reduced when including this term. Additionally, a clear linear trend can be seen in the scale selection histograms for all the three datasets, demonstrating that including a zero-order term in the computational primitives of the GaussDerResNets leads to the selected scale levels still being proportional to the size in the test image data, as expected from theory. (Note that the vertical axes for the test accuracy in the top subfigures use different lower bounds for the different datasets.)
Figure 14: Activation maps of correctly classified images from the rescaled STL-10 dataset, extracted from the final layer of a multi-scale-channel GaussDerResNet with a spatial selection stage based on spatial max pooling and scale selection stage based on average pooling over scales. The activation maps reveal which spatial areas within the image elicit strong responses at the feature channel corresponding to the correct class, for scale channels based on initial scale values , given test samples chosen from size factor 2 test set. Each row depicts results from one of the five chosen classes: (a) “bird”, (b) “car”, (c) “dog”, (d) “monkey” and (e) “ship”. As can be seen from the subfigures, the network typically makes a prediction by identifying characteristic parts of objects, either at fine or coarse scales, which correspond to strong responses at finer scale channels or coarser scale channels, respectively.
Figure 15: Illustration of the main steps in the generation of object hypotheses from deep scale-space maxima extracted from the deep scale-space representations for each class, for the given multi-object test image generated out of Fashion-MNIST samples of different sizes. (a) Original image. (b) Circle representations of the 10 strongest deep scale-space maxima, with the radius proportional to the selected scale for each deep scale-space maximum, and using a relative post-smoothing scale of before the detection of deep scale-space maxima. (c) The result of suppressing overlapping spatial representations within each class. (d) The result of suppressing overlapping spatial representations across all the classes, preserving only the spatial representation having the strongest magnitude value, purely for visualisation purposes. From the resulting circles we can see that the network produces maxima over the objects, no matter the object scale. The colours of the circles indicate the classes of the object hypotheses, with red = “bag” , green = “trouser” , blue = “sneaker” and purple for all the other classes . The underlying image data in (b)–(d) have been brightened to improve clarity.
Figure 16: Illustration of the scale-covariant and scale-invariant properties of the resulting method for generating object hypotheses from deep scale-space maxima from the deep scale-space representation obtained by combining the final layers for the different values of the initial scale parameter for each class. In the top row, images of three objects from the Fashion-MNIST test set have been put at random positions for the size factor 1. In the bottom row, the same objects have been put at other random positions for the size factor 2. As can be seen from the illustrations, the sizes of the circles largely follow the amount of spatial scaling of the input data by the factor of 2 between the first and the second rows. The colours of the circles indicate the classes of the object hypotheses, with red = “ankle boot” , green = “coat” , blue = “bag” and purple for all the other classes . With the exception of the leftmost images, the underlying image data have been brightened to improve clarity.
Figure 17: Illustration of the main steps in the generation of object hypotheses, using three images from the COCO dataset, from deep scale-space maxima obtained by applying a GaussDerResNet based on max pooling over scales, trained on the rescaled STL-10 dataset. Subfigures (a) and (c) both show results for images with a cat and dog seated next each other, while subfigure (b) shows a cat in front of a car. From the resulting circles, we can see that that the network is able to produce strong maxima over multiple objects within real-world images, with the correct classes being the most dominant over the corresponding objects. The colours of the circles indicate the classes of the object hypotheses, with red = “car” , green = “dog” , blue = “cat” and purple for all the other classes .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 18: Scale generalisation curves for the single-scale-channel GaussDerResNets on the rescaled (left) Fashion-MNIST, (middle) CIFAR-10 and (right) STL-10 datasets. Test data with the same size factor 1 as the training data is represented by the shaded grey region. Note that the vertical axes for the test accuracy are in the range 0–100%, unlike the scale generalisation graphs in the main paper.
| rescaled Fashion-MNIST | rescaled CIFAR-10 | rescaled STL-10 | |
|---|---|---|---|
| # network parameters (GaussDerResNets) | 295k | 1.78M | 2.07M |
| # network parameters (DSGaussDerResNets) | 120k | 547k | 669k |
Table 4: Number of network parameters for the multi-scale-channel GaussDerResNets and DSGaussDerResNets employed in our experiments, for the three rescaled datasets.
Figure 19: Comparison between the scale generalisation curves for the multi-scale-channel GaussDerResNets and the depthwise-separable GaussDerResNets (DSGaussDerResNets), with the networks based on scale selection using either max, logsumexp or average pooling over scales, and trained on the rescaled (a) Fashion-MNIST, (b) CIFAR-10 and (c) STL-10 datasets. The training of the models was performed on the training data for the size factor 1, while the evaluation was performed on the corresponding test datasets for all the size factors between 1/2 and 2. As can be seen from the graphs, for the rescaled Fashion-MNIST and STL-10 datasets, the performance for the DSGaussDerResNets is similar to the performance of the GaussDerResNets. For the rescaled CIFAR-10 dataset, however, the performance for larger size factors is lower, probably caused by a less suitable choice of training parameters. (Note that the vertical axes for the test accuracy use different lower bounds for the different datasets.)
Figure 20: Scale selection histograms for the multi-scale-channel DSGaussDerResNets based on max pooling over scales applied to the rescaled (a) Fashion-MNIST, (b) CIFAR-10 and (c) STL-10 datasets. These histograms visualise the relative contribution of each scale channel to the final classification. From these results we can see that, as expected from the theory, the GaussDerResNets based on depthwise-separable convolution operations have very similar scale selection properties as standard GaussDerResNets, with a rather clear linear trend obtained on all the rescaled datasets. This trend demonstrates that the same scale covariance properties hold between the scale channels based on standard or depthwise-separable convolutions, with the selected scale levels being proportional to the size of image features in the testing data.
Figure 21: Scale generalisation curves for the multi-scale-channel networks based on max, logsumexp or average pooling over scales, trained with or without label smoothing during training, for the (left) the rescaled Fashion-MNIST dataset, (middle) the rescaled CIFAR-10 dataset, and (right) the rescaled STL-10 dataset. The networks were then tested on all the size factors between 1/2 and 2, after training on the single size factor 1. The graphs show that using label smoothing as an additional regularisation method during training typically results in better overall performance, while retaining good scale generalisation properties. Note that the vertical axes for the test accuracy use different lower bounds for the different datasets.
Figure 22: Scale selection histograms for the multi-scale-channel GaussDerResNets applied to the rescaled Fashion-MNIST dataset, trained (a)–(c) with label smoothing and trained (d)–(f) without label smoothing, for all the three permutation invariant pooling over scale methods: max, logsumexp and average pooling. The scale selection histograms demonstrate a clear linear trend between the selected scale levels (the scale channels) and the size in the testing data. We can also see that the networks based on label smoothing create more spread out and less diagonal linear trends in the scale selection histograms.
Figure 23: Scale selection histograms for the multi-scale-channel GaussDerResNets applied to the rescaled CIFAR-10 dataset, trained (a)–(c) with label smoothing and trained (d)–(f) without label smoothing, for all the three permutation invariant pooling over scale methods: max, smooth max (logsumexp) and average pooling. The scale selection histograms demonstrate a clear linear trend between the selected scale levels (the scale channels) and the size in the testing data. Once again we can see that the networks based on label smoothing create more spread out and less diagonal linear trends in the scale selection histograms.
Figure 25: Comparison between the scale generalisation curves for the multi-scale-channel GaussDerResNets trained using either pre-training or genuine multi-scale-channel training, with all the networks based on max pooling over scales, on the rescaled (a) Fashion-MNIST, (b) CIFAR-10 and (c) STL-10 datasets. The networks were trained on size factor 1 training data, and then evaluated on rescaled test datasets for each of the size factors between 1/2 and 2. As can be seen from the graphs, pre-training improves the performance for the smaller size factors for all the three rescaled datasets, while for the larger size factors the effect varies. The pre-training improves the performance in some cases or reducing it slightly in other. Note that the vertical axes for the test accuracy use different lower bounds for the different datasets.
| Training Parameters | rescaled Fashion-MNIST | rescaled CIFAR-10 | rescaled STL-10 |
| Number of first stage epochs | 16 | 40 | 50 |
| Number of second stage epochs | 16 | 50 | 70 |
| Batch size during first stage | 128 | 128 | 64 |
| Batch size during second stage | 64 | 45 | 17 |
| Initial learning rate (first stage) | 0.01 | 0.01 | 0.01 |
| Initial learning rate (second stage) | 0.005 | 0.005 | 0.005 |
Table 5: Training configurations for pre-training with single-scale-channel of GaussDerResNets employed in our experiments, for the three rescaled datasets.
Figure 26: Comparison between the zoomed in scale generalisation curves for the multi-scale-channel GaussDerResNets and for the GaussDerResNets extended after training using weight transfer to have additional scale channels that cover a denser scale sampling of the scale levels, with the networks based on max, logsumexp or average pooling over scales, on the rescaled (left) Fashion-MNIST, (middle) CIFAR-10 and (right) STL-10 datasets. The training was performed on training data at size factor 1, and inference was performed on test data for each of the size factors between 1/2 and 2. From the graphs, we can see that in general networks with more scale-channels tend to have marginally improved scale generalisation. Note that the vertical axes for the test accuracy use different lower bounds for the different datasets.
Figure 27: Scale generalisation curves for the rescaled (a) Fashion-MNIST, (b) CIFAR-10, and (c) STL-10 datasets, comparing both the mean accuracy and the error bars for the standard deviation over 5 runs for the GaussDerResNet based on logsumexp pooling over scales, and the Shallow and WideResNet-based DISCO networks. For the rescaled Fashion-MNIST and CIFAR-10 datasets, we additionally compare to the scale generalisation curves of GaussDerNets based on logsumexp pooling over scales, for only a single run (reproduced from Perzanowski and Lindeberg ( 2025d )). The networks were all trained using training data for the size factor 1, and then evaluated on single-scale test sets spanning size factors between 1/2 and 2. From the graphs, we can see that the GaussDerResNets have significantly better scale generalisation properties compared to the DISCO networks, despite using several times fewer parameters. The parameter counts in the legends have been computed in PyTorch.
Figure 28: Scale generalisation curves for the (a) traffic sign and (b) aerial STIR dataset subsets, comparing both the mean accuracy and the error bars for the standard deviation over 5 runs for the multi-scale-channel GaussDerResNets based on logsumexp pooling over scale and multi-scale-channel GaussDerNets based on average pooling (reproduced from Perzanowski and Lindeberg ( 2025d )). These results are compared agains the scale generalisation curves of (i) the SESN, (ii) the DISCO, and (iii) the PixelPool networks, which are some of the best performing models on the STIR benchmark. Three different training regimes are shown for each dataset, indicated by the shaded grey region, with the networks trained on data containing (left) small scale objects (size [17,32]), (middle) medium scale objects (size [33,48]), and (right) large scale objects (size [49,64]). All the networks were evaluated on the test set containing all the possible object sizes, . From the graphs, we see that GaussDerResNets have the best scale generalisation properties among all these networks. The curves for the SESN, DISCO and PixelPool models use results averaged over 25 runs, based directly on results from experiments performed by Altstidl et al. ( 2023 ).
Figure 29: A representative showcase of 10 learned effective filters from an early layer, two intermediate layers from a single residual block and the penultimate layer (specifically the 2nd, 10th, 11th and 17th layers) of a single-scale-channel GaussDerResNet with a zero-order term included in the filter basis, and trained on the rescaled STL-10 dataset. The colour mapping is configured such that zero values appear white, positive values are shown in red, and negative values are represented in blue. The size of these convolution kernels becomes larger in the higher layers, as the consequence of the scale parameter increasing with depth according to Equation (37). We can see that the learned filters are not always DC balanced, and that some of them are similar to different directional derivatives or the Laplacian-of-Gaussian.