NeighborMAE: Exploiting Spatial Dependencies between Neighboring Earth Observation Images in Masked Autoencoders Pretraining

Liang Zeng; Valerio Marsocci; Wufan Zhao; Andrea Nascetti; Maarten Vergauwen

cs.CVMar 3, 2026

NeighborMAE: Exploiting Spatial Dependencies between Neighboring Earth Observation Images in Masked Autoencoders Pretraining

Authors: Liang Zeng, Valerio Marsocci, Wufan Zhao, Andrea Nascetti, Maarten Vergauwen

Organizations: KU Leuven · ESA Φ-lab · HKUST(GZ) · KTH

Abstract

Masked Image Modeling has been one of the most popular self-supervised learning paradigms to learn representations from large-scale, unlabeled Earth Observation images. While incorporating multi-modal and multi-temporal Earth Observation data into Masked Image Modeling has been widely explored, the spatial dependencies between images captured from neighboring areas remains largely overlooked. Since the Earth's surface is continuous, neighboring images are highly related and offer rich contextual information for self-supervised learning. To close this gap, we propose NeighborMAE, which learns spatial dependencies by joint reconstruction of neighboring Earth Observation images. To ensure that the reconstruction remains challenging, we leverage a heuristic strategy to dynamically adjust the mask ratio and the pixel-level loss weight. Experimental results across various pretraining datasets and downstream tasks show that NeighborMAE significantly outperforms existing baselines, underscoring the value of neighboring images in Masked Image Modeling for Earth Observation and the efficacy of our designs.

Figures & tables

Figure 1: NeighborMAE jointly reconstructs masked regions across neighboring EO images by self-attention over all tokens. This design enables the model to capture spatial and other inherent dependencies (e.g., temporal) between neighboring observations.

Figure 2: The overview of NeighborMAE. We sample pairs of neighboring images from datasets based on geographic coordinates. Their relative positions are embedded in a shared coordinate system, and the mask ratio is chosen based on IoU. Neighboring images are jointly reconstructed by MAE, which learns the in-between spatial dependencies by self-attention. The reconstruction loss is weighted by the visibility from the masked input of neighboring images to avoid learning shortcuts.

		fMoW [ 8 ]	FBP [ 38 ]
Data	Input type	Acc.	mIoU
fMoW [ 8 ]	(a) base	58.0 / 76.5	53.5 / 58.2
fMoW [ 8 ]	(b) larger size	58.2 / 76.7	53.8 / 58.6
fMoW [ 8 ]	(c) neighbors	58.7 / 76.8	54.1 / 58.9
fMoW [ 8 ]	(d) MT images	58.2 / 77.0	54.5 / 58.7
fMoW [ 8 ]	(e) MT neighbors	61.7 / 77.7	56.0 / 60.4

Table 2: Using the same input length budget (

2\times

base), pretraining on neighboring images achieved the best performance in both uni-temproal or multi-temporal setting. Moreover, the joint learning spatial and temporal dependencies provides synergistic benefits.

Figure 3: Examples of different ways to extend the view of a base input (a) using a similar number of additional patches . Neighbors introduce diverse spatial variations for learning.

		fMoW [ 8 ]	FBP [ 38 ]
$m_{1}$	$m_{2}$	Acc.	mIoU
0.75	0.75	61.0 / 77.4	55.0 / 59.8
0.75	0.80	61.5 / 77.6	55.7 / 60.3
0.75	0.85	61.7 / 77.7	56.0 / 60.4
0.75	0.90	61.4 / 77.3	55.8 / 60.1
0.80	0.80	60.8 / 77.3	55.1 / 59.4

Table 3: Dynamic mask ratio based on the IoU of neighboring image pairs noticeably improve the performance with lower bound

m_{1}

at 0.75 and upper bound

m_{2}

at 0.85.

	self-	cross-	fMoW [ 8 ]	FBP [ 38 ]
data	visible	visible	Acc.	mIoU
fMoW [ 8 ]		ours	61.7 / 77.7	56.0 / 60.4
fMoW [ 8 ]			61.6 / 77.7	55.8 / 60.5
fMoW [ 8 ]			61.4 / 77.4	55.7 / 60.2
fMoW [ 8 ]			58.6 / 77.0	55.0 / 59.1
Satl. [ 41 ]		ours	52.4 / 75.8	53.2 / 56.8

Table 4: Although our weighted reconstruction loss does not show salient effect on fMoW pretraining with rich temporal changes, it improves pretraining on Satellogic by decreasing the weight for pixels visible from the neighboring image (cross-visible) since images from Satellogic have much fewer temporal changes.

Panel of Figure 4 — Figure 4: fMoW classification performance with different epochs pretraining on fMoW by NeighborMAE, MAE and SatMAE++. NeighborMAE consistently obtains the best performance.

method	batch time (s)	memory per GPU
MAE	0.122	15.5 GB
NeighborMAE	0.134	19.6 GB
SatMAE++	0.381	58.7 GB

Table 5: Batch time and memory usage when training ViT-Base with a total batch size of 1024 distributed on 4 H100 GPUs. NeighborMAE only requires slightly more computational resources than MAE, while multi-scale reconstruction used in SatMAE++ is computationally expensive due to upsampling.

Pretrain Data	fMoW	Satellogic
input size	224
base lr (batch 256)	1.5e-4
actual lr	1.2e-3
lr schedule	cosine decay
weight decay	0.05
optimizer	AdamW

Table 6: Pre-training settings

Model	batch time	wall time
MAE -Large	0.298s	11h 31m 16s
NeighborMAE -Large	0.364s	14h 01m 50s

Table 7: Training time of the main fMoW experiments on 4 H100s.

Experiment setting	linear probing	fine-tuning
input size	224
base lr (batch 256)	1e-3	1e-3
layer-wise lr decay	N/A	0.75
lr schedule	cosine decay
weight decay	0	0.05
optimizer	AdamW

Table 8: Image classification settings used in evaluation protocols.

Experiment setting	frozen backbone	fine-tuning
input size	depends on datasets
base lr (batch 256)	1e-4
layer-wise lr decay	N/A	0.75
lr schedule	cosine decay
weight decay	0	0.05
optimizer	AdamW

Table 9: Semantic segmentation settings used in evaluation protocol.

Panel of Figure 5 — Figure 5: Visualization of the reconstruction of neighboring images from fMoW-RGB. From left to right, we show pairs of neighboring images, masked images, prediction, cross-visible pixels, and the loss weight. Neighboring images from fMoW-RGB usually exhibit significant temporal changes and therefore our loss weighting by cross visibility has less impact.

Panel of Figure 6 — Figure 6: Visualization of the reconstruction of neighboring images from Satellogic-RGB. From left to right, we show pairs of neighboring images, masked images, prediction, cross-visible pixels, and the loss weight. Neighboring images from Satellogic-RGB present fewer changes (fewer and even no revisits from the data). are even identical images. Our loss weighting by input visibility are more effective on Satellogic-RGB to avoid short learning shortcuts.

Panel of Figure 7 — Figure 7: Visualization of the attention of neighboring images. The first row shows the input neighboring images and the second row shows the attention scores with respect the to the patch marked by yellow boundary in the first image. The visualization indicates that NeighborMAE can build spatial relations across neighboring images and learn correlated representations.

Panel of Figure 8 — Figure 8: We show spatial and temporal distributions of fMoW. Pixel values are averaged over overlapping regions. The image sizes are mostly different in fMoW and the images represent specific areas of interest. fMoW features organized temporal sequences captured by multiple EO projects.

Panel of Figure 9 — Figure 9: We show spatial and temporal distributions of Satellogic. Pixel values are averaged over overlapping regions. The image sizes are consistent in Satellogic and the image contents are not semantic-aware. Few or no revisits are provided for specific locations.

Method	$1^{st}$	$2^{nd}$	$3^{rd}$	$4^{th}$	$5^{th}$	mean $\pm$ std
MAE	67.19	66.99	66.71	66.56	65.99	66.69 $\pm$ 0.41
NeighborMAE	69.42	69.33	68.90	68.61	68.08	68.87 $\pm$ 0.49
MAE	78.39	78.22	78.13	78.07	77.91	78.14 $\pm$ 0.16
NeighborMAE	79.53	79.53	79.26	79.24	79.11	79.33 $\pm$ 0.17

Table 10: Performance of 5 independent runs of consecutive SSL pretraining and evaluation on fMoW, using baseline MAE and our NeighborMAE with the main experiment setting in 1 . The upper and lower part shows the accuracy of linear probing and fine-tuning, respectively. We rank the performance from left to right. The results demonstrate that the improvement achieved by NeighborMAE is beyond the deviation caused by randomness.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 238 papers

Far from the Crowd: Scalable Self-Supervised Learning via Geographic Isolation
Aug 20, 2026Daniele Rege Cambrin, Francesco Rossi, Mattia VarileSelf-Supervised Pre-Training Geospatial Representation Learning
AIKO, Corso Stati Uniti 45, Turin, Italy
SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing
Mar 8, 2026Xiaokang Zhang, Bo Li, Chufeng Zhou +2Remote Sensing Image Understanding Remote Sensing
School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence and Automation, Wuhan University of Science and Technology, Wuhan, 430081, China · School of Electronic Information, Wuhan University of Science and Technology, Wuhan, 430081, China +2
The hidden advantage of mask resampling: a theory of masked autoencoders
Oct 1, 2026Jorge Medina Moreira, Lorenzo Bardone, Lenka ZdeborováMasked Autoencoders Masked Language Modeling
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland

Aug 20, 2026J/K move · Enter open · S save

Far from the Crowd: Scalable Self-Supervised Learning via Geographic Isolation

Daniele Rege Cambrin, Francesco Rossi, Mattia Varile

AIKO, Corso Stati Uniti 45, Turin, Italy

Self-Supervised Pre-Training Geospatial Representation Learning Curriculum Learning

Open paper AskPDF arXiv

Self-supervised pretraining on remote sensing imagery typically treats all samples as equally informative, despite large variability in geographic and visual structure. We propose a curriculum learning strategy for self-supervised Earth observation that ranks samples by geographic isolation, a label-free proxy derived entirely from geolocation metadata already present in geospatial datasets, requiring no image decoding, no model feedback, and no manual annotation. Unlike visual complexity proxies, it scales as O(D log D) with dataset size D and is well-defined for both contrastive and reconstructive objectives. We integrate the proposed measure into MoCoV2 and MAE pretraining and evaluate across three downstream tasks from CopernicusBench (BigEarthNet, DFC-2020, LCZ). Our curriculum reaches baseline final-epoch performance using as few as 20% of the training budget (MAE) and at most 40% (MoCo) of the training budget, and improves final downstream performance by up to +5 mAP on BigEarthNet, with gains of 1-5 points across benchmarks, matching visual-complexity curricula while reducing pre-computation cost by more than 140x (4 s vs. 568 s on SSL4EO). A CKA and effective-rank analysis further reveals that curriculum-trained encoders develop higher-dimensional, more uniformly utilized embedding spaces throughout training.

Abstract

Self-Supervised Pre-Training

Explore similar work

Far from the Crowd: Scalable Self-Supervised Learning via Geographic Isolation

SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing

The hidden advantage of mask resampling: a theory of masked autoencoders