Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

Yuki Natsume; Sang-Ho Yun; Chai Kiat Yeo

cs.CVOct 15, 2026

Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

Authors: Yuki Natsume, Sang-Ho Yun, Chai Kiat Yeo

Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · Earth Observatory of Singapore, Nanyang Technological University, Singapore · Asian School of the Environment, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Abstract

Labeled Sentinel-1 Ground Range Detected (S1GRD) datasets remain scarce, limiting supervised deep learning for land-surface remote sensing, while self-supervised pretraining for this domain has been dominated by Vision Transformer (ViT)-based masked image modeling requiring substantial data and compute. We propose SparK-SAR, a sparse convolutional masked pretraining framework built on SparK and ConvNeXt, designed to exploit convolutional inductive biases for S1GRD imagery under constrained compute. SparK-SAR makes three core technical contributions: (i) ConvNeXt-RS, a domain-adapted encoder that halves the stem stride (4x to 2x) to produce 14x14 feature maps at stride 16x, providing 4x more spatial tokens at identical parameter count; (ii) a lightweight ConvNeXt-512 reconstruction decoder with PixelShuffle upsampling that reduces pretraining VRAM by 7.5-8.5% per GPU relative to a UNet-128 baseline while matching its accuracy; and (iii) an empirical characterization of FP16 mixed-precision pretraining instability in the standard SparK UNet-768 decoder on S1GRD data, complemented by a finding that finetuning with layer-wise learning rate decay (LLRD) improves downstream performance of pretrained encoders. On four Copernicus-Bench S1GRD tasks, our finetuned 27.8M-parameter ConvNeXt-RS encoder improves over a non-pretrained baseline by +4.7, +5.0, +5.8, and +1.4 pp on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1 respectively, and outperforms the Copernicus-FM ViT (139M parameters, frozen) on three of four tasks at roughly 5x fewer parameters. A pretraining data-scaling study reveals task-dependent data efficiency: classification saturates with as little as 10-25% of the SSL4EO-S12 v1.1 corpus, while dense prediction continues to improve significantly up to 25-50%. The study also includes a CKA representation analysis and a geographic de-collision experiment.

Figures & tables

Figure 1: The proposed SparK-SAR pretraining framework. The ConvNeXt-RS encoder produces $14{\times}14$ feature maps (stride $16{\times}$ ) from $224{\times}224$ S1GRD inputs. Only visible (unmasked) patches are processed by the encoder via sparse convolutions; the decoder reconstructs the full image including masked regions.

Figure 2: Feature maps of the standard ConvNeXt-Tiny encoder ( $7{\times}7$ , stride $32{\times}$ ). Columns from left to right: VV input, VH input, and Stage 1–4 features. Stage 4 produces $7{\times}7$ maps that lose spatial structure.

Figure 3: Feature maps of the proposed ConvNeXt-RS encoder ( $14{\times}14$ , stride $16{\times}$ ). Stage 4 retains recognizable spatial structure corresponding to scene content, in contrast to Figure 2 .

Figure 4: SparK-SAR reconstruction examples on SSL4EO-S12 v1.1 S1GRD patches. Each row shows (left to right) VV ground truth, VV reconstructed, VH ground truth, and VH reconstructed (60% masking ratio). The reconstructions recover large-scale backscatter structure while appearing smooth relative to ground truth, consistent with the L2 reconstruction objective of MAE-family methods ( He et al., 2022 ) .

Decoder	Hardware	Prec.	VRAM/GPU	Status
UNet-768	A100 80 GB $b=128$	FP16	45,037 MiB	NaN (ep. 5–12)
UNet-768	A100 80 GB $b=128$	BF16	45,091 MiB	Stable
UNet-768	A100 80 GB $b=128$	FP32	80,599 MiB	Stable
UNet-128	V100 32 GB, $b=64$	FP16	17,460 MiB	Stable
ConvNeXt-512	V100 32 GB, $b=64$	FP16	15,974 MiB	Stable
Saving			1,486 MiB (8.5%)

Table 1: Pretraining VRAM usage per GPU for different decoder configurations, hardware, and numerical precision. Our production setup consists of 4

\times

V100 32 GB GPUs. The global batch size is 256 samples across all GPUs.

	SparK (ImageNet)	SparK-SAR (S1GRD)
Dataset size	1.28M images	243,968 locations
Epochs	1,600	50
Batch size	4,096	256
Optimizer	LAMB	AdamW
Max LR	$3.2{\times}10^{-3}$	$10^{-3}$
Encoder output	$7{\times}7$	$14{\times}14$

Table 2: SparK-SAR pretraining recipe compared with the original SparK ImageNet configuration ( Tian et al., 2023 ) .

Decoder	Wall-clock (h)	GPU-hours
UNet-128	8.9	35.6
ConvNeXt-512	7.9	31.6

Table 3: Pretraining compute cost. Both configurations pretrain on 4

\times

NVIDIA V100 (32 GB) GPUs.

Dataset	Task	Metric	Train	Val	Test	Image size
EuroSAT-S1	Classification	OA (%)	16,200	5,400	5,400	$64{\times}64$
BigEarthNet-S1	Multi-label classif.	mAP (%)	11,894	6,117	5,991	$120{\times}120$
DFC2020-S1	Segmentation	mIoU (%)	3,156	986	986	$256{\times}256$
Flood-S1	Change detection	mIoU (%)	3,000	1,000	1,000	$224{\times}224$

Table 4: Copernicus-Bench S1GRD downstream evaluation tasks.

Method	Params	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
		OA (%)	mAP (%)	mIoU (%)	mIoU (%)
Frozen baselines (linear probe / UPerNet head)
Copernicus-FM ViT-B/16 ( Wang et al., 2025b )	139M	87.2 $\pm$ 0.1	77.9 $\pm$ 0.0	52.4 $\pm$ 0.1	77.7 $\pm$ 0.0
SoftCon ViT-B/14 ( Wang et al., 2024 )	86M	83.6 $\pm$ 0.1	78.7 $\pm$ 0.0	52.8 $\pm$ 0.6	77.2 $\pm$ 0.1
CROMA ViT-B/8 ( Fuller et al., 2023 )	86M	83.9 $\pm$ 0.1	70.8 $\pm$ 0.0	52.7 $\pm$ 0.1	77.4 $\pm$ 0.1
DOFA ViT-B/16 ( Xiong et al., 2024 )	111M §	81.7 $\pm$ 0.1	70.5 $\pm$ 0.0	49.7 $\pm$ 0.1	76.0 $\pm$ 0.1

Table 5: Main results on Copernicus-Bench S1GRD tasks. Frozen baselines (top) evaluate a linear probe (classification) or UPerNet head (dense prediction) on frozen pretrained features; SparK-SAR (bottom) is finetuned end-to-end. Because these protocols are not directly comparable, best results are bolded independently within each block.

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
	OA (%)	mAP (%)	mIoU (%)	mIoU (%)
ConvNeXt-RS (frozen)	$76.56{\pm}0.09$	$66.17{\pm}0.03$	$51.94{\pm}0.09$	$\mathbf{78.38{\pm}0.12}$
Frozen ViT baselines (from Table 5 )
Copernicus-FM ViT-B/16	$\mathbf{87.2{\pm}0.1}$	$77.9{\pm}0.0$	$52.4{\pm}0.1$	$77.7{\pm}0.0$
SoftCon ViT-B/14	$83.6{\pm}0.1$	$\mathbf{78.7{\pm}0.0}$	$\mathbf{52.8{\pm}0.6}$	$77.2{\pm}0.1$
CROMA ViT-B/8	$83.9{\pm}0.1$	$70.8{\pm}0.0$	$52.7{\pm}0.1$	$77.4{\pm}0.1$

Table 6: Frozen encoder evaluation of SparK-SAR ConvNeXt-RS (mean

\,\pm\,

std, 5 seeds). The backbone is frozen and only the task head is trained: a linear head for classification tasks and UPerHead ( Xiao et al., 2018 ) for dense prediction tasks. Linear probing is known to disadvantage convolutional architectures relative to ViTs due to the absence of a global summary token ( Tian et al., 2023 ) ; results should be interpreted alongside the full finetuning results in Table 5 .

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
$7{\times}7$ , pretrain $lr{=}10^{-4}$ , UNet-768	$84.57{\pm}0.25$	$72.27{\pm}0.45$	$52.12{\pm}0.03$	$78.67{\pm}0.16$
$7{\times}7$ , pretrain $lr{=}10^{-4}$ , UNet-128	$85.51{\pm}0.11$	$72.45{\pm}0.25$	$52.74{\pm}0.08$	$78.72{\pm}0.21$
$7{\times}7$ , pretrain $lr{=}10^{-3}$ , UNet-128	$87.85{\pm}0.30$	$74.89{\pm}0.38$	$54.22{\pm}0.18$	$79.33{\pm}0.18$
$14{\times}14$ , pretrain $lr{=}10^{-3}$ , UNet-128	$\mathbf{88.11{\pm}0.29}$	$\mathbf{75.47{\pm}0.70}$	$\mathbf{55.59{\pm}0.23}$	$\mathbf{80.11{\pm}0.12}$

Table 7: Output resolution and decoder ablation at pretraining learning rates

10^{-4}

and

10^{-3}

; all encoders were finetuned at

lr{=}10^{-3}

without LLRD (mean

\,\pm\,

std, 5 seeds). All encoders were pretrained with FP16 mixed precision on 4

\times

V100 32 GB GPUs and finetuned in FP32 on V100 GPUs.

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
No LLRD	$88.11\pm 0.29$	$75.47\pm 0.70$	$55.59\pm 0.23$	$80.11\pm 0.12$
LLRD ( $\lambda=0.75$ )	$\mathbf{88.49\pm 0.16}$	$\mathbf{77.03\pm 0.40}$	$\mathbf{56.00\pm 0.08}$	$\mathbf{80.21\pm 0.10}$

Table 8: Effect of LLRD on ConvNeXt-RS finetuning (UNet-128 pretrained, mean

\pm

std, 5 seeds).

$\lambda$	0.50	0.65	0.75 (default)	0.90
DFC2020-S1 (mIoU)	$56.11\pm 0.13$	$56.06\pm 0.10$	$56.00\pm 0.08$	$55.81\pm 0.09$ *
Flood-S1 (mIoU)	$80.09\pm 0.24$	$80.18\pm 0.22$	$80.21\pm 0.10$	$80.01\pm 0.15$

Table 9: Sensitivity of downstream performance to the LLRD decay factor

\lambda

, evaluated on DFC2020-S1 and Flood-S1 (mIoU, %; mean

\pm

std over 5 seeds). The BEiT-inherited default used throughout the paper is

\lambda=0.75

; this column is the production run (V100), whereas the other columns were run on an NVIDIA RTX A6000 GPU.

LLRD	Scheduler	Fusion	Loss	Flood-S1 mIoU (%)
No	OneCycleLR	Subtraction	CE	$79.99\pm 0.14$
Yes	OneCycleLR	Subtraction	CE	$\mathbf{80.11\pm 0.15}$
Yes	CosineAnneal	Subtraction	CE	$80.10\pm 0.07$
Yes	OneCycleLR	Concatenation	CE + TV	$80.20\pm 0.12$
Yes	OneCycleLR	Concatenation	Tversky	$75.75\pm 1.78$
Yes	OneCycleLR	Concatenation	CE	$80.53\pm 0.15$

Table 10: Flood-S1 ablation: finetuning configuration variants (ConvNeXt-RS with ConvNeXt-512 pretrained encoder; mean

\pm

std over 5 seeds (42, 123, 456, 789, 1337),

lr{=}10^{-3}

, NVIDIA RTX A6000 GPU). CE: cross-entropy; TV: Tversky loss (

\alpha{=}0.3

\beta{=}0.7

); Focal: focal loss (

\gamma{=}2

). Subtraction:

f_{\text{post}}-f_{\text{pre}}

; Concatenation:

[f_{\text{pre}};f_{\text{post}}]

. Baseline (bold) is the primary configuration used throughout this paper (its V100 production run in Table 5 gives

80.11\pm 0.10

). All rows use OneCycleLR except the CosineAnneal row.

Figure 5: Pretraining data scaling: finetuning accuracy vs. pretraining data fraction (mean $\,\pm\,$ std, 5 seeds), corresponding to Table 11 .

Subset	Tiles	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
0% (random)	0	$83.29{\pm}0.54^{\ast}$	$72.10{\pm}0.20^{\ast}$	$50.20{\pm}0.17^{\ast}$	$78.03{\pm}0.45^{\ast}$
1%	2,439	$85.12{\pm}0.42^{\ast}$	$73.67{\pm}0.33^{\ast}$	$52.58{\pm}0.14^{\ast}$	$79.49{\pm}0.12^{\ast}$
5%	12,198	$87.58{\pm}0.38^{\ast}$	$76.12{\pm}0.62$	$54.59{\pm}0.07^{\ast}$	$79.93{\pm}0.08^{\ast}$
10%	24,396	$88.12{\pm}0.27$	$76.58{\pm}0.40$	$55.10{\pm}0.12^{\ast}$	$80.11{\pm}0.12$
25%	60,992	$88.34{\pm}0.19$	$76.75{\pm}0.29$	$55.55{\pm}0.13^{\ast}$	$80.09{\pm}0.11$
50%	121,984	$88.68{\pm}0.29$	$76.47{\pm}0.32^{\ast}$	$55.84{\pm}0.16$	$80.20{\pm}0.22$

Table 11: Pretraining data scaling results (ConvNeXt-RS, UNet-128, LLRD, mean

\,\pm\,

std, 5 seeds).

\ast

indicates statistically significant difference from 100% (paired two-tailed t-test, BH FDR correction across 28 comparisons,

p_{\mathrm{BH}}<0.05

Figure 6: Label efficiency: finetuning accuracy vs. labeled-data fraction, pretrained (PT) vs. random initialization (No-PT) (mean $\,\pm\,$ std, 5 seeds), corresponding to Table 12 . Dashed line marks the 100%-label No-PT level.

	EuroSAT-S1 (OA%)		BigEarthNet-S1 (mAP%)		DFC2020-S1 (mIoU%)		Flood-S1 (mIoU%)
Labels	No-PT	PT	No-PT	PT	No-PT	PT	No-PT	PT
1%	47.94 $\pm$ 0.49	59.54 $\pm$ 3.65	46.41 $\pm$ 1.10	50.01 $\pm$ 1.52	30.19 $\pm$ 0.70	38.05 $\pm$ 1.39	48.17 $\pm$ 1.36	54.43 $\pm$ 3.38
5%	67.07 $\pm$ 1.31	79.56 $\pm$ 0.83	56.99 $\pm$ 0.40	62.67 $\pm$ 0.36	35.59 $\pm$ 0.87	46.10 $\pm$ 0.33	68.64 $\pm$ 0.69	74.01 $\pm$ 0.60
10%	73.53 $\pm$ 1.13	82.34 $\pm$ 0.80	59.67 $\pm$ 0.55	66.90 $\pm$ 0.87	39.18 $\pm$ 0.55	49.13 $\pm$ 0.50	71.23 $\pm$ 0.28	76.02 $\pm$ 0.46
25%	78.62 $\pm$ 0.70	84.98 $\pm$ 0.35	65.12 $\pm$ 0.30	71.39 $\pm$ 0.62	43.90 $\pm$ 0.70	52.41 $\pm$ 0.29	73.96 $\pm$ 0.55	77.70 $\pm$ 0.36
50%	81.19 $\pm$ 0.44	87.10 $\pm$ 0.30	68.16 $\pm$ 0.20	73.97 $\pm$ 0.18	47.25 $\pm$ 0.13	54.34 $\pm$ 0.17	75.94 $\pm$ 1.04	78.52 $\pm$ 0.32

Table 12: Label efficiency: finetuning accuracy vs. labeled-data fraction, pretrained (PT) vs. random initialization (No-PT), UNet-128 decoder, finetuned using LLRD, 5 seeds. All pretrained

>

random initialized gaps are statistically significant (paired

t

-test, BH-corrected across 20 comparisons,

p_{BH}<0.05

; max

p_{BH}=0.011

Figure 7: (a) CKA similarity to fully pretrained (100%) encoder vs. pretraining data fraction (Stage 3 post-GAP features, unbiased HSIC, Z-score standardization). (b) Pairwise CKA heatmap for DFC2020-S1. Values near the diagonal confirm that representations converge as data increases, with saturation consistent with downstream performance.

Task	Original	De-collided	$\Delta$
EuroSAT-S1	88.49 $\pm$ 0.16	88.46 $\pm$ 0.31	$-0.03$
BigEarthNet-S1	77.03 $\pm$ 0.40	77.08 $\pm$ 0.47	$+0.05$
DFC2020-S1	56.00 $\pm$ 0.08	56.02 $\pm$ 0.10	$+0.02$
Flood-S1	80.21 $\pm$ 0.10	80.30 $\pm$ 0.18	$+0.09$

Table 13: Full experimental de-collision: SparK-SAR re-pretrained on SSL4EO-S12 v1.1 with all overlapping tiles removed, re-finetuned on all four tasks (mean

\,\pm\,

std, 5 seeds).

Figure 8: DFC2020-S1 qualitative comparison across four representative scenes (rows). Columns from left to right: VV input, VH input, ground truth, and predictions from the standard-resolution ConvNeXt encoder (pretrained, no LLRD; Table 7 ), the ConvNeXt-RS encoder without pretraining (no LLRD; Table 5 ), and our finetuned ConvNeXt-RS encoder (pretrained, LLRD; Table 5 ). LLRD was evaluated only for ConvNeXt-RS.

Figure 9: Flood-S1 qualitative comparison across four representative scenes (rows). Columns from left to right: pre-flood VV, post-flood VV, ground truth, and predictions from the standard-resolution ConvNeXt encoder (pretrained, no LLRD; Table 7 ), the ConvNeXt-RS encoder without pretraining (no LLRD; Table 5 ), and our finetuned ConvNeXt-RS encoder (pretrained, LLRD; Table 5 ). LLRD was evaluated only for ConvNeXt-RS.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 251 papers

Jul 25, 2026

SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models

Masked image modeling has become a dominant paradigm for SAR pre-training, yet the design of the reconstruction target remains fundamentally unsettled. This article argues that a SAR pre-training target should satisfy two conditions to produce transferable representations: (i) physics-grounded stability, i.e., approximate invariance of the target operator to multiplicative speckle inherent in coherent imaging; and (ii) semantic scale compatibility, i.e., coverage of the heterogeneous spatial scales that downstream tasks demand. These two conditions are individually achievable but jointly difficult: physics-grounded stability favors fixed operators, while semantic scale compatibility favors data-driven composition. To this end, SARATR-X-v2 reconciles both within a single design. The target is constructed through fixed structural extractors spanning six receptive fields, from blind-spot local aggregation to directional log-ratio region contrast, and fused via learnable weights into one unified supervision signal for masked reconstruction. On twelve SAR benchmarks across classification, detection, and segmentation, SARATR-X-v2 achieves state-of-the-art transfer performance. Under synthetic speckle variation, the proposed target reduces perturbation drift in the learned representation by nearly two orders of magnitude relative to pixel-space supervision. Taken together, these results establish physics-grounded stability and semantic scale compatibility as a principled framework for pre-training target design under coherent imaging, and suggest that effective SAR pre-training is not about reconstructing more signal, but about reconstructing the right structural target.

Weijie Li, Yafei Song, Yongxiang Liu +7

College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China

Self-Supervised Pre-Training Synthetic Aperture Radar Masked Image Modeling

Ask PDF

Jul 13, 2026

Self-supervised training for high-resolution close-range multispectral remote sensing imagery

Although self-supervised learning (SSL) offers a promising way to reduce annotation effort in close-range remote sensing, its effectiveness for high-resolution multispectral unmanned aerial vehicle (UAV) imagery remains underexplored due to limited data. This study evaluated SSL pretraining for precision agriculture using cm-scale multispectral drone imagery collected across multiple sensors, years, and regions. Transformer-based encoders were pretrained with Momentum Contrast v3 (MoCo-v3) and Masked Autoencoders on a harmonized dataset combining msuav500K with newly collected multi-year UAV imagery from agricultural fields in Finland. Pretraining used four spectral bands (Green, Red, Red-Edge, Near-Infrared) for cross-sensor compatibility. The models were evaluated on crop-weed semantic segmentation using the WeedMap dataset with 5--100% training data. The following two subsets served as downstream tasks: Task A (Germany, RedEdge-M), where all pretrained models were compared under partial and full fine-tuning, and Task B (Switzerland, Sequoia), where the best encoder from Task A was assessed. Our Swin Transformer pretrained with MoCo-v3 achieved the strongest performance on both tasks, surpassing the Swin Transformer model of Doornbos et al. pretrained on a pre-release of msuav500K. Our pretrained Swin Transformer further demonstrated cross-sensor and cross-region generalization. We additionally provide a public multi-year multispectral UAV dataset from Finland to support future research.

Leon-Friedrich Thomas, Mikael Änäkkälä, Antti Lajunen

University of Helsinki, Department of Agricultural Sciences, P.O. Box 28, Fi-00014 Helsinki, Finland · University of Helsinki, Department of Forest Sciences, P.O. Box 27, Fi-00014 Helsinki, Finland

Agricultural Image Analysis Remote Sensing Image Segmentation Agricultural Remote Sensing Self-Supervised Pre-Training UAV Remote Sensing

Ask PDF

May 27, 2026

A Deep Learning Iterative Framework for Sentinel-1 Stripmap Enhancement Based on Azimuth Doppler Decomposition

Synthetic Aperture Radar (SAR) imagery enables all-weather, day-and-night Earth observation; however, it remains difficult to interpret due to speckle noise and other intrinsic imaging artifacts. Sentinel-1 (S1) constitutes one of the most widely used spaceborne SAR missions, offering systematic global coverage, high temporal resolution, dual-polarization imaging, and free data availability. Among S1 modes, Stripmap (SM) provides the highest resolution, yet speckle noise and spatial constraints often hinder applications requiring finer spatial detail. This motivates the need for effective image enhancement strategies. In this work, we propose a self-supervised enhancement framework for S1 SM imagery based on azimuth subaperture decomposition. The method exploits the physical consistency between subaperture reconstructions and the corresponding full-aperture image to generate paired training data without external sensors, simulated ground truth, or multi-temporal stacks. The proposed framework integrates single- and multi-frame learning and incorporates an iterative inference scheme that progressively refines image quality. Experiments on real S1 SM data show that the proposed approach consistently outperforms the widely adopted self-supervised deep learning baseline MERLIN, in terms of PSNR and SSIM, while MERLIN attains higher ENL, highlighting a trade-off between structural fidelity and speckle smoothing. Overall, the results demonstrate that subaperture-based supervision provides a physically grounded, reproducible, and operationally viable approach for SAR image enhancement using S1 data. It is worth noting that the proposed approach can be extended to other SAR platforms, polarizations, and acquisition modes.

Juan Francisco Amieva, Christian Ayala, Roberto Del Prete +1

Tracasa Instrumental S.L. · European Space Agency · Public University of Navarre

Image Enhancement Self-Supervised Learning Remote Sensing Synthetic Aperture Radar Image Denoising

Ask PDF

Abstract

Self-Supervised Pre-Training

Explore similar work

SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models

Self-supervised training for high-resolution close-range multispectral remote sensing imagery

A Deep Learning Iterative Framework for Sentinel-1 Stripmap Enhancement Based on Azimuth Doppler Decomposition