Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

Yuki Natsume; Sang-Ho Yun; Chai Kiat Yeo

cs.CVOct 15, 2026

Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

Authors: Yuki Natsume, Sang-Ho Yun, Chai Kiat Yeo

Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · Earth Observatory of Singapore, Nanyang Technological University, Singapore · Asian School of the Environment, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Abstract

Labeled Sentinel-1 Ground Range Detected (S1GRD) datasets remain scarce, limiting supervised deep learning for land-surface remote sensing, while self-supervised pretraining for this domain has been dominated by Vision Transformer (ViT)-based masked image modeling requiring substantial data and compute. We propose SparK-SAR, a sparse convolutional masked pretraining framework built on SparK and ConvNeXt, designed to exploit convolutional inductive biases for S1GRD imagery under constrained compute. SparK-SAR makes three core technical contributions: (i) ConvNeXt-RS, a domain-adapted encoder that halves the stem stride (4x to 2x) to produce 14x14 feature maps at stride 16x, providing 4x more spatial tokens at identical parameter count; (ii) a lightweight ConvNeXt-512 reconstruction decoder with PixelShuffle upsampling that reduces pretraining VRAM by 7.5-8.5% per GPU relative to a UNet-128 baseline while matching its accuracy; and (iii) an empirical characterization of FP16 mixed-precision pretraining instability in the standard SparK UNet-768 decoder on S1GRD data, complemented by a finding that finetuning with layer-wise learning rate decay (LLRD) improves downstream performance of pretrained encoders. On four Copernicus-Bench S1GRD tasks, our finetuned 27.8M-parameter ConvNeXt-RS encoder improves over a non-pretrained baseline by +4.7, +5.0, +5.8, and +1.4 pp on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1 respectively, and outperforms the Copernicus-FM ViT (139M parameters, frozen) on three of four tasks at roughly 5x fewer parameters. A pretraining data-scaling study reveals task-dependent data efficiency: classification saturates with as little as 10-25% of the SSL4EO-S12 v1.1 corpus, while dense prediction continues to improve significantly up to 25-50%. The study also includes a CKA representation analysis and a geographic de-collision experiment.

Figures & tables

Figure 1: The proposed SparK-SAR pretraining framework. The ConvNeXt-RS encoder produces $14{\times}14$ feature maps (stride $16{\times}$ ) from $224{\times}224$ S1GRD inputs. Only visible (unmasked) patches are processed by the encoder via sparse convolutions; the decoder reconstructs the full image including masked regions.

Figure 2: Feature maps of the standard ConvNeXt-Tiny encoder ( $7{\times}7$ , stride $32{\times}$ ). Columns from left to right: VV input, VH input, and Stage 1–4 features. Stage 4 produces $7{\times}7$ maps that lose spatial structure.

Figure 3: Feature maps of the proposed ConvNeXt-RS encoder ( $14{\times}14$ , stride $16{\times}$ ). Stage 4 retains recognizable spatial structure corresponding to scene content, in contrast to Figure 2 .

Figure 4: SparK-SAR reconstruction examples on SSL4EO-S12 v1.1 S1GRD patches. Each row shows (left to right) VV ground truth, VV reconstructed, VH ground truth, and VH reconstructed (60% masking ratio). The reconstructions recover large-scale backscatter structure while appearing smooth relative to ground truth, consistent with the L2 reconstruction objective of MAE-family methods ( He et al., 2022 ) .

Decoder	Hardware	Prec.	VRAM/GPU	Status
UNet-768	A100 80 GB $b=128$	FP16	45,037 MiB	NaN (ep. 5–12)
UNet-768	A100 80 GB $b=128$	BF16	45,091 MiB	Stable
UNet-768	A100 80 GB $b=128$	FP32	80,599 MiB	Stable
UNet-128	V100 32 GB, $b=64$	FP16	17,460 MiB	Stable
ConvNeXt-512	V100 32 GB, $b=64$	FP16	15,974 MiB	Stable
Saving			1,486 MiB (8.5%)

Table 1: Pretraining VRAM usage per GPU for different decoder configurations, hardware, and numerical precision. Our production setup consists of 4

\times

V100 32 GB GPUs. The global batch size is 256 samples across all GPUs.

	SparK (ImageNet)	SparK-SAR (S1GRD)
Dataset size	1.28M images	243,968 locations
Epochs	1,600	50
Batch size	4,096	256
Optimizer	LAMB	AdamW
Max LR	$3.2{\times}10^{-3}$	$10^{-3}$
Encoder output	$7{\times}7$	$14{\times}14$

Table 2: SparK-SAR pretraining recipe compared with the original SparK ImageNet configuration ( Tian et al., 2023 ) .

Decoder	Wall-clock (h)	GPU-hours
UNet-128	8.9	35.6
ConvNeXt-512	7.9	31.6

Table 3: Pretraining compute cost. Both configurations pretrain on 4

\times

NVIDIA V100 (32 GB) GPUs.

Dataset	Task	Metric	Train	Val	Test	Image size
EuroSAT-S1	Classification	OA (%)	16,200	5,400	5,400	$64{\times}64$
BigEarthNet-S1	Multi-label classif.	mAP (%)	11,894	6,117	5,991	$120{\times}120$
DFC2020-S1	Segmentation	mIoU (%)	3,156	986	986	$256{\times}256$
Flood-S1	Change detection	mIoU (%)	3,000	1,000	1,000	$224{\times}224$

Table 4: Copernicus-Bench S1GRD downstream evaluation tasks.

Method	Params	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
		OA (%)	mAP (%)	mIoU (%)	mIoU (%)
Frozen baselines (linear probe / UPerNet head)
Copernicus-FM ViT-B/16 ( Wang et al., 2025b )	139M	87.2 $\pm$ 0.1	77.9 $\pm$ 0.0	52.4 $\pm$ 0.1	77.7 $\pm$ 0.0
SoftCon ViT-B/14 ( Wang et al., 2024 )	86M	83.6 $\pm$ 0.1	78.7 $\pm$ 0.0	52.8 $\pm$ 0.6	77.2 $\pm$ 0.1
CROMA ViT-B/8 ( Fuller et al., 2023 )	86M	83.9 $\pm$ 0.1	70.8 $\pm$ 0.0	52.7 $\pm$ 0.1	77.4 $\pm$ 0.1
DOFA ViT-B/16 ( Xiong et al., 2024 )	111M §	81.7 $\pm$ 0.1	70.5 $\pm$ 0.0	49.7 $\pm$ 0.1	76.0 $\pm$ 0.1

Table 5: Main results on Copernicus-Bench S1GRD tasks. Frozen baselines (top) evaluate a linear probe (classification) or UPerNet head (dense prediction) on frozen pretrained features; SparK-SAR (bottom) is finetuned end-to-end. Because these protocols are not directly comparable, best results are bolded independently within each block.

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
	OA (%)	mAP (%)	mIoU (%)	mIoU (%)
ConvNeXt-RS (frozen)	$76.56{\pm}0.09$	$66.17{\pm}0.03$	$51.94{\pm}0.09$	$\mathbf{78.38{\pm}0.12}$
Frozen ViT baselines (from Table 5 )
Copernicus-FM ViT-B/16	$\mathbf{87.2{\pm}0.1}$	$77.9{\pm}0.0$	$52.4{\pm}0.1$	$77.7{\pm}0.0$
SoftCon ViT-B/14	$83.6{\pm}0.1$	$\mathbf{78.7{\pm}0.0}$	$\mathbf{52.8{\pm}0.6}$	$77.2{\pm}0.1$
CROMA ViT-B/8	$83.9{\pm}0.1$	$70.8{\pm}0.0$	$52.7{\pm}0.1$	$77.4{\pm}0.1$

Table 6: Frozen encoder evaluation of SparK-SAR ConvNeXt-RS (mean

\,\pm\,

std, 5 seeds). The backbone is frozen and only the task head is trained: a linear head for classification tasks and UPerHead ( Xiao et al., 2018 ) for dense prediction tasks. Linear probing is known to disadvantage convolutional architectures relative to ViTs due to the absence of a global summary token ( Tian et al., 2023 ) ; results should be interpreted alongside the full finetuning results in Table 5 .

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
$7{\times}7$ , pretrain $lr{=}10^{-4}$ , UNet-768	$84.57{\pm}0.25$	$72.27{\pm}0.45$	$52.12{\pm}0.03$	$78.67{\pm}0.16$
$7{\times}7$ , pretrain $lr{=}10^{-4}$ , UNet-128	$85.51{\pm}0.11$	$72.45{\pm}0.25$	$52.74{\pm}0.08$	$78.72{\pm}0.21$
$7{\times}7$ , pretrain $lr{=}10^{-3}$ , UNet-128	$87.85{\pm}0.30$	$74.89{\pm}0.38$	$54.22{\pm}0.18$	$79.33{\pm}0.18$
$14{\times}14$ , pretrain $lr{=}10^{-3}$ , UNet-128	$\mathbf{88.11{\pm}0.29}$	$\mathbf{75.47{\pm}0.70}$	$\mathbf{55.59{\pm}0.23}$	$\mathbf{80.11{\pm}0.12}$

Table 7: Output resolution and decoder ablation at pretraining learning rates

10^{-4}

and

10^{-3}

; all encoders were finetuned at

lr{=}10^{-3}

without LLRD (mean

\,\pm\,

std, 5 seeds). All encoders were pretrained with FP16 mixed precision on 4

\times

V100 32 GB GPUs and finetuned in FP32 on V100 GPUs.

Config	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
No LLRD	$88.11\pm 0.29$	$75.47\pm 0.70$	$55.59\pm 0.23$	$80.11\pm 0.12$
LLRD ( $\lambda=0.75$ )	$\mathbf{88.49\pm 0.16}$	$\mathbf{77.03\pm 0.40}$	$\mathbf{56.00\pm 0.08}$	$\mathbf{80.21\pm 0.10}$

Table 8: Effect of LLRD on ConvNeXt-RS finetuning (UNet-128 pretrained, mean

\pm

std, 5 seeds).

$\lambda$	0.50	0.65	0.75 (default)	0.90
DFC2020-S1 (mIoU)	$56.11\pm 0.13$	$56.06\pm 0.10$	$56.00\pm 0.08$	$55.81\pm 0.09$ *
Flood-S1 (mIoU)	$80.09\pm 0.24$	$80.18\pm 0.22$	$80.21\pm 0.10$	$80.01\pm 0.15$

Table 9: Sensitivity of downstream performance to the LLRD decay factor

\lambda

, evaluated on DFC2020-S1 and Flood-S1 (mIoU, %; mean

\pm

std over 5 seeds). The BEiT-inherited default used throughout the paper is

\lambda=0.75

; this column is the production run (V100), whereas the other columns were run on an NVIDIA RTX A6000 GPU.

LLRD	Scheduler	Fusion	Loss	Flood-S1 mIoU (%)
No	OneCycleLR	Subtraction	CE	$79.99\pm 0.14$
Yes	OneCycleLR	Subtraction	CE	$\mathbf{80.11\pm 0.15}$
Yes	CosineAnneal	Subtraction	CE	$80.10\pm 0.07$
Yes	OneCycleLR	Concatenation	CE + TV	$80.20\pm 0.12$
Yes	OneCycleLR	Concatenation	Tversky	$75.75\pm 1.78$
Yes	OneCycleLR	Concatenation	CE	$80.53\pm 0.15$

Table 10: Flood-S1 ablation: finetuning configuration variants (ConvNeXt-RS with ConvNeXt-512 pretrained encoder; mean

\pm

std over 5 seeds (42, 123, 456, 789, 1337),

lr{=}10^{-3}

, NVIDIA RTX A6000 GPU). CE: cross-entropy; TV: Tversky loss (

\alpha{=}0.3

\beta{=}0.7

); Focal: focal loss (

\gamma{=}2

). Subtraction:

f_{\text{post}}-f_{\text{pre}}

; Concatenation:

[f_{\text{pre}};f_{\text{post}}]

. Baseline (bold) is the primary configuration used throughout this paper (its V100 production run in Table 5 gives

80.11\pm 0.10

). All rows use OneCycleLR except the CosineAnneal row.

Figure 5: Pretraining data scaling: finetuning accuracy vs. pretraining data fraction (mean $\,\pm\,$ std, 5 seeds), corresponding to Table 11 .

Subset	Tiles	EuroSAT-S1	BigEarthNet-S1	DFC2020-S1	Flood-S1
0% (random)	0	$83.29{\pm}0.54^{\ast}$	$72.10{\pm}0.20^{\ast}$	$50.20{\pm}0.17^{\ast}$	$78.03{\pm}0.45^{\ast}$
1%	2,439	$85.12{\pm}0.42^{\ast}$	$73.67{\pm}0.33^{\ast}$	$52.58{\pm}0.14^{\ast}$	$79.49{\pm}0.12^{\ast}$
5%	12,198	$87.58{\pm}0.38^{\ast}$	$76.12{\pm}0.62$	$54.59{\pm}0.07^{\ast}$	$79.93{\pm}0.08^{\ast}$
10%	24,396	$88.12{\pm}0.27$	$76.58{\pm}0.40$	$55.10{\pm}0.12^{\ast}$	$80.11{\pm}0.12$
25%	60,992	$88.34{\pm}0.19$	$76.75{\pm}0.29$	$55.55{\pm}0.13^{\ast}$	$80.09{\pm}0.11$
50%	121,984	$88.68{\pm}0.29$	$76.47{\pm}0.32^{\ast}$	$55.84{\pm}0.16$	$80.20{\pm}0.22$

Table 11: Pretraining data scaling results (ConvNeXt-RS, UNet-128, LLRD, mean

\,\pm\,

std, 5 seeds).

\ast

indicates statistically significant difference from 100% (paired two-tailed t-test, BH FDR correction across 28 comparisons,

p_{\mathrm{BH}}<0.05

Figure 6: Label efficiency: finetuning accuracy vs. labeled-data fraction, pretrained (PT) vs. random initialization (No-PT) (mean $\,\pm\,$ std, 5 seeds), corresponding to Table 12 . Dashed line marks the 100%-label No-PT level.

	EuroSAT-S1 (OA%)		BigEarthNet-S1 (mAP%)		DFC2020-S1 (mIoU%)		Flood-S1 (mIoU%)
Labels	No-PT	PT	No-PT	PT	No-PT	PT	No-PT	PT
1%	47.94 $\pm$ 0.49	59.54 $\pm$ 3.65	46.41 $\pm$ 1.10	50.01 $\pm$ 1.52	30.19 $\pm$ 0.70	38.05 $\pm$ 1.39	48.17 $\pm$ 1.36	54.43 $\pm$ 3.38
5%	67.07 $\pm$ 1.31	79.56 $\pm$ 0.83	56.99 $\pm$ 0.40	62.67 $\pm$ 0.36	35.59 $\pm$ 0.87	46.10 $\pm$ 0.33	68.64 $\pm$ 0.69	74.01 $\pm$ 0.60
10%	73.53 $\pm$ 1.13	82.34 $\pm$ 0.80	59.67 $\pm$ 0.55	66.90 $\pm$ 0.87	39.18 $\pm$ 0.55	49.13 $\pm$ 0.50	71.23 $\pm$ 0.28	76.02 $\pm$ 0.46
25%	78.62 $\pm$ 0.70	84.98 $\pm$ 0.35	65.12 $\pm$ 0.30	71.39 $\pm$ 0.62	43.90 $\pm$ 0.70	52.41 $\pm$ 0.29	73.96 $\pm$ 0.55	77.70 $\pm$ 0.36
50%	81.19 $\pm$ 0.44	87.10 $\pm$ 0.30	68.16 $\pm$ 0.20	73.97 $\pm$ 0.18	47.25 $\pm$ 0.13	54.34 $\pm$ 0.17	75.94 $\pm$ 1.04	78.52 $\pm$ 0.32

Table 12: Label efficiency: finetuning accuracy vs. labeled-data fraction, pretrained (PT) vs. random initialization (No-PT), UNet-128 decoder, finetuned using LLRD, 5 seeds. All pretrained

>

random initialized gaps are statistically significant (paired

t

-test, BH-corrected across 20 comparisons,

p_{BH}<0.05

; max

p_{BH}=0.011

Figure 7: (a) CKA similarity to fully pretrained (100%) encoder vs. pretraining data fraction (Stage 3 post-GAP features, unbiased HSIC, Z-score standardization). (b) Pairwise CKA heatmap for DFC2020-S1. Values near the diagonal confirm that representations converge as data increases, with saturation consistent with downstream performance.

Task	Original	De-collided	$\Delta$
EuroSAT-S1	88.49 $\pm$ 0.16	88.46 $\pm$ 0.31	$-0.03$
BigEarthNet-S1	77.03 $\pm$ 0.40	77.08 $\pm$ 0.47	$+0.05$
DFC2020-S1	56.00 $\pm$ 0.08	56.02 $\pm$ 0.10	$+0.02$
Flood-S1	80.21 $\pm$ 0.10	80.30 $\pm$ 0.18	$+0.09$

Table 13: Full experimental de-collision: SparK-SAR re-pretrained on SSL4EO-S12 v1.1 with all overlapping tiles removed, re-finetuned on all four tasks (mean

\,\pm\,

std, 5 seeds).

Figure 8: DFC2020-S1 qualitative comparison across four representative scenes (rows). Columns from left to right: VV input, VH input, ground truth, and predictions from the standard-resolution ConvNeXt encoder (pretrained, no LLRD; Table 7 ), the ConvNeXt-RS encoder without pretraining (no LLRD; Table 5 ), and our finetuned ConvNeXt-RS encoder (pretrained, LLRD; Table 5 ). LLRD was evaluated only for ConvNeXt-RS.

Figure 9: Flood-S1 qualitative comparison across four representative scenes (rows). Columns from left to right: pre-flood VV, post-flood VV, ground truth, and predictions from the standard-resolution ConvNeXt encoder (pretrained, no LLRD; Table 7 ), the ConvNeXt-RS encoder without pretraining (no LLRD; Table 5 ), and our finetuned ConvNeXt-RS encoder (pretrained, LLRD; Table 5 ). LLRD was evaluated only for ConvNeXt-RS.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 251 papers

SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models
Jul 25, 2026Weijie Li, Yafei Song, Yongxiang Liu +7Self-Supervised Pre-Training Synthetic Aperture Radar
College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China
Self-supervised training for high-resolution close-range multispectral remote sensing imagery
Jul 13, 2026Leon-Friedrich Thomas, Mikael Änäkkälä, Antti LajunenAgricultural Image Analysis Remote Sensing Image Segmentation
University of Helsinki, Department of Agricultural Sciences, P.O. Box 28, Fi-00014 Helsinki, Finland · University of Helsinki, Department of Forest Sciences, P.O. Box 27, Fi-00014 Helsinki, Finland
A Deep Learning Iterative Framework for Sentinel-1 Stripmap Enhancement Based on Azimuth Doppler Decomposition
May 27, 2026Juan Francisco Amieva, Christian Ayala, Roberto Del Prete +1Image Enhancement Self-Supervised Learning
Tracasa Instrumental S.L. · European Space Agency · Public University of Navarre

Jul 25, 2026J/K move · Enter open · S save

SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models

Weijie Li, Yafei Song, Yongxiang Liu, Bowen Peng+6

College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China

Self-Supervised Pre-Training Synthetic Aperture Radar Masked Image Modeling

Open paper AskPDF arXiv

Masked image modeling has become a dominant paradigm for SAR pre-training, yet the design of the reconstruction target remains fundamentally unsettled. This article argues that a SAR pre-training target should satisfy two conditions to produce transferable representations: (i) physics-grounded stability, i.e., approximate invariance of the target operator to multiplicative speckle inherent in coherent imaging; and (ii) semantic scale compatibility, i.e., coverage of the heterogeneous spatial scales that downstream tasks demand. These two conditions are individually achievable but jointly difficult: physics-grounded stability favors fixed operators, while semantic scale compatibility favors data-driven composition. To this end, SARATR-X-v2 reconciles both within a single design. The target is constructed through fixed structural extractors spanning six receptive fields, from blind-spot local aggregation to directional log-ratio region contrast, and fused via learnable weights into one unified supervision signal for masked reconstruction. On twelve SAR benchmarks across classification, detection, and segmentation, SARATR-X-v2 achieves state-of-the-art transfer performance. Under synthetic speckle variation, the proposed target reduces perturbation drift in the learned representation by nearly two orders of magnitude relative to pixel-space supervision. Taken together, these results establish physics-grounded stability and semantic scale compatibility as a principled framework for pre-training target design under coherent imaging, and suggest that effective SAR pre-training is not about reconstructing more signal, but about reconstructing the right structural target.

Abstract

Self-Supervised Pre-Training

Explore similar work

SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models

Self-supervised training for high-resolution close-range multispectral remote sensing imagery

A Deep Learning Iterative Framework for Sentinel-1 Stripmap Enhancement Based on Azimuth Doppler Decomposition