Learnable downsampling and spectral-domain filter parameterizations have each made convolutional networks more flexible, but their interaction has not been systematically examined. We bridge this gap by examining convolutional kernels learned in the spatial and spectral domains, together with learned downsampling ratios under a cropped resizing implementation. We find that spectral and spatial kernels respond differently to learned resolution allocation, with larger gains for the spectral kernel. We observe this consistently on three datasets (EuroSAT, CIFAR-10 and STL-10) and at two model sizes (differences in accuracy gain of +0.013 to +0.028 on held-out test sets). Part of this difference reflects the greater gain of the spectral kernel from retaining more spatial area, but the difference remains positive when total retained area is matched. This pattern is observed across datasets and model sizes, although its magnitude and dependence on the complexity budget vary with model size and the resizing implementation.
Figures & tables
Figure 1: Gain from making S learnable, by filter type, under the cropped implementation (learnable resizing with feature maps cropped where the mask falls below 0.01). Held-out test accuracy relative to the fixed schedule at p⋆=0.12 ; points are individual seeds, horizontal bars are means and vertical bars are 95% confidence intervals ( n=10 ). The spectral filters gain more in all four settings.
Gain ( crop )
Δgain
spatial
spectral
crop
ramp
soft
EuroSAT-S
+0.009
+0.022
+0.013∗
+0.006
+0.020∗
EuroSAT
+0.004
+0.021
+0.016∗
+0.012∗
+0.022∗
CIFAR-10
−0.007
+0.011
+0.018∗
+0.002
+0.038∗
STL-10
+0.003
+0.031
+0.028∗
+0.008
+0.049∗
Table 1: Gain from making S learnable at p⋆=0.12 , held-out test accuracy ( n=10 ). The first two columns are the gains under crop; the last three are Δgain under each implementation. EuroSAT-S uses the smaller models ( 2.1×104 spatial and 2.5×104 spectral parameters); all other rows use approximately 5×104 parameters. ∗ : uncorrected p<0.05 ; all four crop comparisons remain significant after Holm correction.
Control
Δgain ( p )
Equal area, EuroSAT-S
+0.006 ( 0.08 )
Equal area, EuroSAT
+0.013 ( 0.004 )
Equal area, CIFAR-10
+0.015 ( 0.03 )
Equal area, STL-10
+0.009 ( 0.13 )
Spatial baseline, F=8 ( 12.7 k params)
+0.020 ( 0.002 )
Spatial baseline, F=19 ( 25.3 k params)
+0.023 ( 0.002 )
Table 2: Controls at p⋆=0.12 . Top: crop compared with a fixed schedule of matched retained area ( p⋆=0.25 ), validation. Middle: additional controls, test unless marked “val.”. Bottom: a fixed high-ratio schedule ( s1/s2=2.06 ) against the default fixed schedule, test.
Figure 2: Δgain on EuroSAT under soft . (a) Across seven budgets at two model sizes ( n=20 per point, validation). (b) Across model size at p⋆=0.12 ( n=10 , validation). Error bars are 95% confidence intervals.
Figure 3: Learned allocation and kernel size on EuroSAT at p⋆=0.12 . (a) Mean adjacent-stage resolution ratios under soft over 10 seeds (separate runs), with the spectral kernel at stage 1 or 2; error bars are one standard deviation. (b) Effective area of the SpecMLP filters (640 per configuration) on the layer’s own grid at two energy thresholds; boxes show quartiles, whiskers the 5th–95th percentiles.
Convolutional neural networks (CNNs) learn spatial and temporal patterns using shared filters, with applications from image recognition to scientific prediction.
Is the same data equally useful to every learner? The question is whether the value of a training sample belongs to the sample or arises in its interaction with the learner. We test it with a selection experiment. Under a per-class budget, two kinds of samples compete for the slots: easy samples, which a short training run learns first, and coverage samples, which spread over the class in feature space. Easy samples win at small budgets and coverage samples at large ones, and the budget at which the winner flips, the crossover m0, can be measured. We ask whether m0 is set by the data or by the learner. To separate the two, we freeze the selected subsets and change only the learner that trains on them. On low-resolution ImageNet-100, doubling the width of a ResNet-18 moves the crossover from 57 to 85 samples per class on the same subsets. Enlarging the input grid from 32 to 64 pixels at fixed image information moves it from 85 to 57, and a single stride change reproduces or removes this shift. The grid shift shrinks as width grows. Under the native ImageNet-1k protocol, a 4x range of width moves the boundary by at most a few samples per class. On a Vision Transformer, coverage wins at every measured budget, even with easy subsets that work on the convolutional learner. The budget at which one kind of sample stops being the more useful kind therefore belongs to the learner, and the preferred regime of a selection strategy has to be read against the learner that will use the data.
Hybrid CNN-Transformer architectures achieve strong results in image super-resolution, but scaling attention windows or convolution kernels significantly increases computational cost, limiting deployment on resource-constrained devices. We present UCAN, a lightweight network that unifies convolution and attention to expand the effective receptive field efficiently. UCAN combines window-based spatial attention with a Hedgehog Attention mechanism to model both local texture and long-range dependencies, and introduces a distillation-based large-kernel module to preserve high-frequency structure without heavy computation. In addition, we employ cross-layer parameter sharing to further reduce complexity. On Manga109 (4×), UCAN-L achieves 31.63 dB PSNR with only 48.4G MACs, surpassing recent lightweight models. On BSDS100, UCAN attains 27.79 dB, outperforming methods with significantly larger models. Extensive experiments show that UCAN achieves a superior trade-off between accuracy, efficiency, and scalability, making it well-suited for practical high-resolution image restoration.
Cao Thien Tan, Phan Thi Thu Trang, Do Nghiem Duc +3
Ho Chi Minh City Open University · AI Tech Lab, Ho Chi Minh City University of Technology · Code Mely AI Research Team +3
Diffusion transformers deliver strong image generation, but their training cost grows superlinearly with resolution. Recent work justifies training or sampling at reduced resolution on a spectral premise: at high noise, a downscaled latent preserves almost the full surviving signal. Whether a downscaled step also preserves the native training gradient signal, however, has remained unresolved. We reduce how that signal changes under downscaling to two terms: a noise-dependent term governed by the downscale ratio, which decays at high noise as the spectral premise predicts, and a σ-independent floor governed by the target grid's absolute token count, carried by the compute graph itself and removed by no noise level. The measured (route, σ) map corroborates the account and uncovers structure the spectral picture cannot express: on the 1024->768 route, a window (0.65 < σ< 0.95), predicted by no spectral criterion at any tolerance, where the downscaled gradient stays within a small margin of the native one. Training LoRA adapters with downscaled steps restricted to the routes and noise windows the map validates reduces training time by 14.6% at a fixed step budget while remaining near-native in weight space. Code is available at https://github.com/sorryhyun/anima_lora.